scala - Spark 加载决策树数据 - 更改 LabelledPoint 中的标签

Question

我尝试在 Spark 中做决策树的示例 https://spark.apache.org/docs/latest/mllib-decision-tree.html

我已经从http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#a1a下载了a1a数据集

数据集采用 LIBSVM 格式，其中两个类的标签为 +1.0 和 -1.0 当我尝试时

import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils

// Load and parse the data file.
val data = MLUtils.loadLibSVMFile(sc, "/user/cloudera/testDT/a1a.t")
// Split the data into training and test sets (30% held out for testing)
val splits = data.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))

// Train a DecisionTree model.
//  Empty categoricalFeaturesInfo indicates all features are continuous.
val numClasses = 2
val categoricalFeaturesInfo = Map[Int, Int]()
val impurity = "gini"
val maxDepth = 5
val maxBins = 32

val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,
 |   impurity, maxDepth, maxBins)

我得到:

java.lang.IllegalArgumentException: GiniAggregator given label -1.0 but requires label is non-negative.

所以我尝试将标签-1.0更改为0.0。我尝试过类似的东西

def changeLabel(a: org.apache.spark.mllib.regression.LabeledPoint) =
 { if (a.label == -1.0) {a.label = 0.0}  }

我在哪里遇到错误:

reassignment to val

所以我的问题是:如何更改数据的标签？或者是否有解决方法，使 DecisionTree.trainClassifier() 可以处理带有负标签的数据？

Answer 1

TL;DR 您不能放弃 Product 类的值参数，即使可以(声明为 var)，您绝不修改 Spark 中的数据。

怎么样:

def changeLabel(a: org.apache.spark.mllib.regression.LabeledPoint) = 
  if (a.label == -1.0) a.copy(label = 0.0) else a

scala> changeLabel(LabeledPoint(-1.0, Vectors.dense(1.0, 2.0, 3.0)))
res1: org.apache.spark.mllib.regression.LabeledPoint = (0.0,[1.0,2.0,3.0])

scala> changeLabel(LabeledPoint(1.0, Vectors.dense(1.0, 2.0, 3.0)))
res2: org.apache.spark.mllib.regression.LabeledPoint = (1.0,[1.0,2.0,3.0])