我有大约 5000 行和 950 列的 csv 文件。首先,我将它加载到 DataFrame:
val data = sqlContext.read
.format(csvFormat)
.option("header", "true")
.option("inferSchema", "true")
.load(file)
.cache()
之后我搜索所有字符串列
val featuresToIndex = data.schema
.filter(_.dataType == StringType)
.map(field => field.name)
并希望将它们编入索引。为此,我为每个字符串列创建了索引器
val stringIndexers = featuresToIndex.map(colName =>
new StringIndexer()
.setInputCol(colName)
.setOutputCol(colName + "Indexed"))
并创建管道
val pipeline = new Pipeline().setStages(stringIndexers.toArray)
但是当我尝试使用此管道转换我的初始数据框时
val indexedDf = pipeline.fit(data).transform(data)
我收到 StackOverflowError
16/07/05 16:55:12 INFO DAGScheduler: Job 4 finished: countByValue at StringIndexer.scala:86, took 7.882774 s
Exception in thread "main" java.lang.StackOverflowError
at scala.collection.immutable.Set$Set1.contains(Set.scala:84)
at scala.collection.immutable.Set$Set1.$plus(Set.scala:86)
at scala.collection.immutable.Set$Set1.$plus(Set.scala:81)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:22)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:20)
at scala.collection.generic.Growable$class.loop$1(Growable.scala:53)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:57)
at scala.collection.mutable.SetBuilder.$plus$plus$eq(SetBuilder.scala:20)
at scala.collection.TraversableLike$class.to(TraversableLike.scala:590)
at scala.collection.AbstractTraversable.to(Traversable.scala:104)
at scala.collection.TraversableOnce$class.toSet(TraversableOnce.scala:304)
at scala.collection.AbstractTraversable.toSet(Traversable.scala:104)
at org.apache.spark.sql.catalyst.trees.TreeNode.containsChild$lzycompute(TreeNode.scala:86)
at org.apache.spark.sql.catalyst.trees.TreeNode.containsChild(TreeNode.scala:86)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:280)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
...
我做错了什么? 谢谢。
最佳答案
很可能没有足够的内存来保存所有堆栈帧。我在训练 RandomForestModel 时经历过类似的事情。对我有用的解决方法是使用附加参数运行我的驱动程序应用程序(即 Web 服务):
-XX:ThreadStackSize=81920 -Dspark.executor.extraJavaOptions='-XX:ThreadStackSize=81920'
关于java - Apache Spark : StackOverflowError when trying to indexing string columns,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/38206166/