我正在使用 spark 版本 2.2.0 和 Python 2.7。我正在使用 pyspark 连接 BigSQL 并尝试检索数据。以下是我使用的代码
import cPickle as cpick
import numpy as np
import pandas as pd
import time
import sys
from pyspark.sql.session import SparkSession
spark = SparkSession.builder.getOrCreate()
spark_train_df = spark.read.jdbc("jdbc:db2://BigSQL URL:Port:sslConnection=true;","Schema.Table",
properties={"user": "my userid",
"password": "password",
'driver' : 'com.ibm.db2.jcc.DB2Driver'})
spark_train_df.registerTempTable('data_table')
# query to get columns necessary to create indexes
sql = "select * FROM data_table"
train_df = spark.sql(sql)
cmr_dict = { 'date': time.strftime('%a, %b %d, %Y'),
'description': '`cmrs` contains data from data_table',
'cmrs': train_df}
with open('cmrs.pkl', mode='wb') as fp:
cpick.dump(cmr_dict, fp, cpick.HIGHEST_PROTOCOL)
运行代码后我收到错误消息
Py4JError: An error occurred while calling o79.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:272)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
最佳答案
不可能 pickle Spark 分布式对象。这些只是 JVM 结构的代理，更不用说它们不包含任何数据(仅描述计算)。
如果你想 pickle 数据，收集并序列化结果
关于python - Pyspark错误+方法__getnewargs__([])不存在，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/46734044/