我有两对 RDD 可以说
RDD1 : [(1,a),(2,b),(3,c)]
RDD2 : [(1,d),(2,e),(3,f)]
现在我使用 join 加入这些 RDD
RDD3 = RDD1.join(RDD2);
我已经用下面的代码显示了 RDD3 中的元素
for(Tuple2<Integer,Tuple2<String,String>> tuple : RDD3.collect())
System.out.println(tuple._1()+":"+tuple._2()._1()+","+tuple._2()._2());
我看到过奇怪的结果,例如
5:b,e
4:a,d
6:c,f
正如我所料
1:a,d
1:b,e
1:c,f
有没有办法获得像上面这样的期望输出? 或者我错误地解释了 RDD 的行为?请提出建议
编辑:
实际上我正在读取这样的数据
JavaDoubleRDD data1 = sc.parallelizeDoubles(Arrays.asList(45.25,22.15,33.24));
JavaDoubleRDD data2 = sc.parallelizeDoubles(Arrays.asList(23.45,19.35,12.45));
然后
JavaPairRDD<Double,Double> lat1 = data1.cartesian(data1);
JavaRDD<Double> lat2 = lat1.map(new Function<Tuple2<Double,Double>,Double>() {
@Override
public Double call(Tuple2<Double,Double> t) {
return Math.pow(t._1()-t._2(),2);
}
});
//flag and flag1 are static variables initially equal to 1
JavaPairRDD<Integer,Double> lat3 = lat2.mapToPair(new PairFunction<Double,Integer,Double>() {
@Override
public Tuple2<Integer,Double> call(Double d) {
return new Tuple2<Integer,Double>(flag++,d);
}
});
System.out.println("Latitude values display");
for(Tuple2<?,?> tuple : lat3.collect()) {
System.out.println(tuple._1()+":"+tuple._2());
}
JavaPairRDD<Double,Double> long1 = data2.cartesian(data2);
JavaRDD<Double> long2 = long1.map(new Function<Tuple2<Double,Double>,Double>() {
@Override
public Double call(Tuple2<Double,Double> t) {
return Math.pow(t._1()-t._2(),2);
}
});
JavaPairRDD<Integer,Double> long3 = long2.mapToPair(new PairFunction<Double,Integer,Double>() {
@Override
public Tuple2<Integer,Double> call(Double d ) {
return new Tuple2<Integer,Double>(flag1++,d);
}
});
System.out.println("Longitude values display");
for(Tuple2<?,?> tuple : long3.collect()) {
System.out.println(tuple._1()+":"+tuple._2());
}
System.out.println("latitude and longitude values join");
JavaPairRDD<Integer,Tuple2<Double,Double>> weightmatrix1 = lat3.join(long3);
System.out.println("Weightmatrix1 Display");
for(Tuple2<?,Tuple2<?,?>> tuple : weightmatrix1.collect()) {
System.out.println(tuple._1()+":"+tuple._2()._1()+","+tuple._2()._2());
}
所以我正在做的是根据纬度和经度值计算权重矩阵
最佳答案
当我这样做时:
scala> val rdd1 = sc.parallelize(Array((1,"a"),(2,"b"),(3,"c")))
scala> val rdd2 = sc.parallelize(Array((1,"d"),(2,"e"),(3,"f")))
scala> val rdd3 = rdd1.join(rdd2)
scala> rdd3.toArray.foreach(println(_))
我一直得到:
(1,(a,d))
(2,(b,e))
(3,(c,f))
关于java - RDD join : After joining two different pair RDDs, 结果RDD键值和顺序发生了变化?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/30008572/