我最近在我的 Windows 机器(使用 Java 8)上安装了 Spark 2.3，并且能够通过 Jupyter Notebooks (Python 3) 运行它。
突然它停止工作 - 我在 Notebook 中尝试实例化 SparkContext 时遇到以下错误:
from pyspark import SparkContext
sc = pyspark.SparkContext()
在每个单元格一行的基础上拆分代码表明它是导致它的第二行。
这似乎纯粹是笔记本问题，因为我仍然能够通过命令行使用“spark-submit”执行 .py 文件。
知道如何解决吗？
-------------------------------------------------
Py4JError Traceback (most recent call last)
<ipython-input-78-57590c71cf44> in <module>()
1 from pyspark import SparkContext
----> 2 sc = pyspark.SparkContext()
~\Anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
116 try:
117 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
--> 118 conf, jsc, profiler_cls)
119 except:
120 # If an error occurs, clean up in order to allow future SparkContext creation:
~\Anaconda3\lib\site-packages\pyspark\context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
186 self._accumulatorServer = accumulators._start_update_server()
187 (host, port) = self._accumulatorServer.server_address
--> 188 self._javaAccumulator = self._jvm.PythonAccumulatorV2(host, port)
189 self._jsc.sc().register(self._javaAccumulator)
190
~\Anaconda3\lib\site-packages\py4j\java_gateway.py in __call__(self, *args)
1523 answer = self._gateway_client.send_command(command)
1524 return_value = get_return_value(
-> 1525 answer, self._gateway_client, None, self._fqn)
1526
1527 for temp_arg in temp_args:
~\Anaconda3\lib\site-packages\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
--> 332 format(target_id, ".", name, value))
333 else:
334 raise Py4JError(
Py4JError: An error occurred while calling None.org.apache.spark.api.python.PythonAccumulatorV2. Trace:
py4j.Py4JException: Constructor org.apache.spark.api.python.PythonAccumulatorV2([class java.lang.String, class java.lang.Integer]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
at py4j.Gateway.invoke(Gateway.java:237)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Unknown Source)
最佳答案
我也遇到了同样的问题。这基本上是由于 pyspark/spark 版本冲突，所以请检查“子版本”是否也匹配，例如v2.3.1 与 v2.3.2
您可以使用:
pip install pyspark==2.3.x to get the right version
关于python-3.x - "PythonAccumulatorV2 does not exist"- 在 Jupyter Notebook 中运行 SparkContext() 时，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/53421623/