我有一个 pandas 数据框 my_df
,my_df.dtypes
给我们:
ts int64
fieldA object
fieldB object
fieldC object
fieldD object
fieldE object
dtype: object
然后我尝试通过以下操作将 pandas 数据框 my_df
转换为 spark 数据框:
spark_my_df = sc.createDataFrame(my_df)
但是,我遇到了以下错误:
ValueErrorTraceback (most recent call last)
<ipython-input-29-d4c9bb41bb1e> in <module>()
----> 1 spark_my_df = sc.createDataFrame(my_df)
2 spark_my_df.take(20)
/usr/local/spark-latest/python/pyspark/sql/session.py in createDataFrame(self, data, schema, samplingRatio)
520 rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
521 else:
--> 522 rdd, schema = self._createFromLocal(map(prepare, data), schema)
523 jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
524 jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
/usr/local/spark-latest/python/pyspark/sql/session.py in _createFromLocal(self, data, schema)
384
385 if schema is None or isinstance(schema, (list, tuple)):
--> 386 struct = self._inferSchemaFromList(data)
387 if isinstance(schema, (list, tuple)):
388 for i, name in enumerate(schema):
/usr/local/spark-latest/python/pyspark/sql/session.py in _inferSchemaFromList(self, data)
318 schema = reduce(_merge_type, map(_infer_schema, data))
319 if _has_nulltype(schema):
--> 320 raise ValueError("Some of types cannot be determined after inferring")
321 return schema
322
ValueError: Some of types cannot be determined after inferring
有谁知道上面的错误是什么意思?谢谢!
最佳答案
为了推断字段类型,PySpark 查看每个字段中的非无记录。如果一个字段只有 None 记录,PySpark 无法推断类型并将引发该错误。
手动定义模式将解决问题
>>> from pyspark.sql.types import StructType, StructField, StringType
>>> schema = StructType([StructField("foo", StringType(), True)])
>>> df = spark.createDataFrame([[None]], schema=schema)
>>> df.show()
+----+
|foo |
+----+
|null|
+----+
关于python - pyspark:ValueError:推断后无法确定某些类型,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/40517553/