我在 Spark 中加载了具有以下模式的 DataFrame:
电子邮件, 名, 姓, 订单号
我如何通过电子邮件对其进行分组,计算每组中的记录并返回具有此架构的 DataFrame:
电子邮件, 名, 姓, 订单数
最佳答案
这是在 Scala 中实现的一种方法:
val df = sc.parallelize(Seq(("a","b","c",1),("a","b","c",2),("x","xb","xc",3),("y","yb","yc",4),("x","xb","xc",5))).toDF("email","first_name","last_name","order_id")
df.registerTempTable("df")
sqlContext.sql("select * from (select email, count(*) as order_count from df group by email ) d1 join df d2 on d1.email = d2.email")
在 Java 中,考虑到您已经创建了 DataFrame,它实际上是相同的代码:
DataFrame results = sqlContext.sql("select * from (select email, count(*) as order_count from df group by email ) d1 join df d2 on d1.email = d2.email");
尽管如此,甚至认为这是直接的解决方案,但我认为这是一种不好的做法,因为您的代码将难以维护和发展。更清洁的解决方案是:
DataFrame email_count = df.groupBy("email").count();
DataFrame results2 = email_count.join(df, email_count.col("email").equalTo(df.col("email"))).drop(df.col("email"));
关于java - 使用 Apache Spark 和 Java 对 DataFrame 进行分组和聚合?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34996378/