根据我的last question我必须为我独特的 JVM 定义 Multiple SparkContext。
我用下一种方式(使用 Java)做到了:
SparkConf conf = new SparkConf();
conf.setAppName("Spark MultipleContest Test");
conf.set("spark.driver.allowMultipleContexts", "true");
conf.setMaster("local");
之后我创建下一个源代码:
SparkContext sc = new SparkContext(conf);
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
和后面的代码:
JavaSparkContext ctx = new JavaSparkContext(conf);
JavaRDD<Row> testRDD = ctx.parallelize(AllList);
代码执行后我收到下一条错误信息:
16/01/19 15:21:08 WARN SparkContext: Multiple running SparkContexts detected in the same JVM!
org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:
org.apache.spark.SparkContext.<init>(SparkContext.scala:81)
test.MLlib.BinarryClassification.main(BinaryClassification.java:41)
at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2083)
at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2065)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2065)
at org.apache.spark.SparkContext$.setActiveContext(SparkContext.scala:2151)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:2023)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at test.MLlib.BinarryClassification.main(BinaryClassification.java:105)
数字 41
和 105
是行,其中两个对象都是在 Java 代码中定义的。我的问题是,如果我已经使用了 set
-method,是否可以在同一个 JVM 上执行多个 SparkContext 以及如何执行?
最佳答案
您确定需要将 JavaSparkContext 作为单独的上下文吗?您提到的上一个问题并没有这么说。如果您已经有一个 Spark 上下文,您可以从中创建一个新的 JavaSparkContext,而不是创建一个单独的上下文:
SparkConf conf = new SparkConf();
conf.setAppName("Spark MultipleContest Test");
conf.set("spark.driver.allowMultipleContexts", "true");
conf.setMaster("local");
SparkContext sc = new SparkContext(conf);
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
//Create a Java Context which is the same as the scala one under the hood
JavaSparkContext.fromSparkContext(sc)
关于java - 在同一 JVM 中检测到多个 SparkContext,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34879414/