当从新的 Spark shell REPL session 运行以下方法时,一切正常。但是,当我尝试编译包含此方法的类时,出现以下错误
Error:(21, 50) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
^
Error:(22, 65) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
^
Error:(23, 47) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val test = ratings.filter(x => x._1 >= 8).values.persist
^
在这两种情况下,我都使用 Spark 1.0.1 代码本身如下。
def createDataset(ratings: RDD[Tuple2[Long,Rating]]): List[RDD[Rating]] = {
val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
val test = ratings.filter(x => x._1 >= 8).values.persist
val numTraining = training.count
val numValidation = validation.count
val numTest = test.count
println(" Number Of Training ::: " + numTraining + " numValidation ::: " + numValidation + " ::: " + numTest)
List(training,validation,test)
}
它取自 MLLib 教程(稍作修改),不知道出了什么问题。
最佳答案
您需要在代码中包含此行
import org.apache.spark.SparkContext._
这会将隐式转换导入 PairRDDFunctions
这就是您可以调用值
的原因。 Spark REPL 会为您导入此内容,这就是您在解释器中看不到错误的原因。具体来说,SparkContext
中的这个函数会执行转换。
implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
(implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null) = {
new PairRDDFunctions(rdd)
}
关于scala - REPL 返回 RDD 值,但 SBT 无法编译,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/25002088/