我对 ML 和 Spark ML 都很陌生,我正在尝试使用带有 Spark ML 的神经网络制作预测模型,但是当我调用 .transform
方法时出现此错误在我学到的模型上。问题是由使用 OneHotEncoder 引起的,因为没有它一切正常。
我试过将 OneHotEncoder 从管道中取出。
我的问题是:如何使用 OneHotEncoder 而不会出现此错误?
java.lang.IllegalArgumentException: requirement failed: A & B Dimension mismatch!
at scala.Predef$.require(Predef.scala:224) at org.apache.spark.ml.ann.BreezeUtil$.dgemm(BreezeUtil.scala:41) at
org.apache.spark.ml.ann.AffineLayerModel.eval(Layer.scala:163) at
org.apache.spark.ml.ann.FeedForwardModel.forward(Layer.scala:482) at
org.apache.spark.ml.ann.FeedForwardModel.predict(Layer.scala:529)
我的代码:
test_pandas_df = pd.read_csv(
'/home/piotrek/ml/adults/adult.test', names=header, skipinitialspace=True)
train_pandas_df = pd.read_csv(
'/home/piotrek/ml/adults/adult.data', names=header, skipinitialspace=True)
train_df = sqlContext.createDataFrame(train_pandas_df)
test_df = sqlContext.createDataFrame(test_pandas_df)
joined = train_df.union(test_df)
assembler = VectorAssembler().setInputCols(features).setOutputCol("features")
label_indexer = StringIndexer().setInputCol(
"label").setOutputCol("label_index")
label_indexer_fit = [label_indexer.fit(joined)]
string_indexers = [StringIndexer().setInputCol(
name).setOutputCol(name + "_index").fit(joined) for name in categorical_feats]
one_hot_pipeline = Pipeline().setStages([OneHotEncoder().setInputCol(
name + '_index').setOutputCol(name + '_one_hot') for name in categorical_feats])
mlp = MultilayerPerceptronClassifier().setLabelCol(label_indexer.getOutputCol()).setFeaturesCol(
assembler.getOutputCol()).setLayers([len(features), 20, 10, 2]).setSeed(42L).setBlockSize(1000).setMaxIter(500)
pipeline = Pipeline().setStages(label_indexer_fit
+ string_indexers + [one_hot_pipeline] + [assembler, mlp])
model = pipeline.fit(train_df)
# compute accuracy on the test set
result = model.transform(test_df)
## FAILS ON RESULT
predictionAndLabels = result.select("prediction", "label_index")
evaluator = MulticlassClassificationEvaluator(labelCol="label_index")
print "-------------------------------"
print("Test set accuracy = " + str(evaluator.evaluate(predictionAndLabels)))
print "-------------------------------"
谢谢!
最佳答案
layers
Param
在你的模型中不正确:
setLayers([len(features), 20, 10, 2])
第一层应反射(reflect)输入特征的数量,通常与编码前的原始列数不同。
如果您不知道预先的特征总数,您可以将特征提取和模型训练分开。伪代码:
feature_pipeline_model = (Pipeline()
.setStages(...) # Only feature extraction
.fit(train_df))
train_df_features = feature_pipeline_model.transform(train_df)
layers = [
train_df_features.schema["features"].metadata["ml_attr"]["num_attrs"],
20, 10, 2
]
关于python - Spark ML 中的维度不匹配错误,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/42301111/