scala - 如何根据两列对 Spark RDD 进行排序

Question

我有以下 RDD 数据集:

ABC [G4, G3, G1]    3
FFF [G5, G4, G3]    3
CDE [G5,G4,G3,G2]   4
XYZ [G4, G3]    2

需要先按最后一列 desc 排序，如果最后一列相同，则按第一个元组项 desc 顺序排序。预期结果是

CDE [G5,G4,G3,G2]   4
FFF [G5, G4, G3]    3
ABC [G4, G3, G1]    3
XYZ [G4, G3]    2

提前致谢。

Answer 1

您可以使用sortBy:

rdd.sortBy(r => (r._3, r._2(0)), false)

上面的r._3代表最后一列，r._2(0)代表第二列的第一个元素(这是一个数组) 、false 指定顺序应为降序。请记住，由于洗牌，排序是一项昂贵的操作。

更新

如果我们假设您从一对 rdd 开始，那么这是一个可重现的示例:

/// Generate data
val rdd = sc.parallelize(Seq(("ABC","G4"),("ABC","G3"),
                             ("ABC","G1"),("FFF","G5"),
                             ("FFF","G4"),("FFF","G3"),
                             ("CDE","G5"),("CDE","G4"),                             
                             ("CDE","G3"),("CDE","G2"),
                             ("XYZ","G4"),("XYZ","G3")))

/// Put values in a list and calculate its size
val rdd_new = rdd.groupByKey.mapValues(_.toList).map(x => (x._1, x._2, x._2.size))

/// Now this works
rdd_new.sortBy(r => (r._3, r._2(0)), false).collect()
/// Array[(String, List[String], Int)] = Array((CDE,List(G5, G4, G3, G2),4), (FFF,List(G5, G4, G3),3), (ABC,List(G4, G3, G1),3), (XYZ,List(G4, G3),2))