hadoop - 映射减少事件时，有没有办法打开Unix命令？

Question

我正在尝试在配置单元查询中获取UNIX输出。
例如，以下查询不起作用:

select transform ('') 
using 'pwd' 
as syspath

但是此查询有效:

select transform ('') 
using 'hive -e "select 10 as col1"' 
as col1

我如何启用UNIX命令或bash脚本作为映射减少作业以使其在hive中可用的输出？

提前致谢!

Answer 1

count the number of characters from a file

您为什么要为此使用Hive？ Spark更加灵活。

val charCount = spark.read.textLines("path/to/file.txt")
    .flatMap(line => line.toList())
    .map(char => (char, 1)) // This is literally just wordcount, now
    .reduceByKey(_ + _)
    .map((char, count) => count)
    .sum() // something like this ... 
println(charCount.collect()(0))