apache-spark - 如何为二元分类选择平衡采样？

Question

这是我的代码，从配置单元加载数据，并进行样本平衡:

// Load SubSet Data
val dataList = DataLoader.loadSubTrainTestData(hiveContext.sql(sampleDataHql))

// Split Data to Train and Test
val data = dataList.randomSplit(Array(0.7, 0.3), seed = 11L)

// Random balance train data
val sampleCount = data(0).map(rec => (rec.label, 1)).reduceByKey(_ + _)

val positiveSample = data(0).filter(_.label == 1).cache()
val positiveSize = positiveSample.count()

val negativeSample = data(0).filter(_.label == 0).cache()
val negativeSize = negativeSample.count()

// Build train data
val trainData = positiveSample ++
negativeSample.sample(withReplacement = false, 1.0 * positiveSize.toFloat / negativeSize, System.nanoTime())

// Data size
val trainDataSize = positiveSize + negativeSize
val testDataSize = trainDataSize * 3.0 / 7.0

我计算trainDataSize和testDataSize来评估模型置信度

Answer 1

好吧，我还没有测试过这段代码，但它应该是这样的:

val data: RDD[LabeledPoint] = ???

val fractions: Map[Double, Double] = Map(0.0 -> 0.5, 1.0 -> 0.5)
val sampledData: RDD[LabeledPoint] = data
  .keyBy(_.label)
  .sampleByKeyExact(false, fractions)  // Optionally with seed
  .values

您可以将 LabeledPoint 转换为 PairRDD，而不是使用您希望使用的分数应用sampleByKeyExact。