我想使用 Spark 和 Java8 从 Dataframe 中获取多列的不同值的计数
输入数据帧 - 需要为动态列编写代码 - 列可能稍后添加
+----+----+----+
|Col1|Col2|Col3|
+----+----+----+
|A1|Y|B2|Y|C3|Y|
|A1|Y|B2|N|C3|Y|
|A1|Y|B2|Y|C3|N|
+----+----+----+
输出日期帧
+--------+---------------------+--------------------+
|Col1 | Col2 | Col3 |
+--------+---------------------+--------------------+
|A1|Y - 3| B2|Y - 2 & B2|N - 1 | C3|Y - 3 & C3|N -1 |
+----+----+----+----+----+----+----+----+----+------+
最佳答案
也许这可以帮助你一点,在scala中使用rdd,但在java中应该非常相似。
val df = Seq(("a", "a", "a"), ("a", "b", "c"), ("b", "b", "c")).toDF("Col1","Col2","Col3")
df.show()
val ok = df.rdd.map(s => {
var arr = new Array[(String, String)](s.size)
for (i <- 0 to s.size - 1) {
arr(i) = (s.getString(i), s.schema.fieldNames(i))
}
arr
}).map(s => {
for (i <- s) yield ((i._2, i._1), 1)
}).flatMap(s => s)
.reduceByKey(_ + _)
.map(s => (s._1._1, s._1._2 + "=" + s._2))
.reduceByKey(_ +","+ _)
ok.foreach(println(_))
+----+----+----+
|Col1|Col2|Col3|
+----+----+----+
| a| a| a|
| a| b| c|
| b| b| c|
+----+----+----+
(Col1,a=2,b=1)
(Col2,b=2,a=1)
(Col3,a=1,c=2)
关于java - 使用 Spark 和 Java8 从 Dataframe 获取多列的不同值计数,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58394575/