我在连接两个数据帧时得到重复项,其中一个键是小数,另一个是字符串。似乎 Spark 正在将小数转换为字符串,从而产生科学记数法表达式,但随后以小数形式显示原始结果就好了。我通过直接转换为字符串找到了解决方法,但这似乎很危险,因为在没有警告的情况下创建了重复项。 这是一个错误吗?我如何检测这种情况何时发生?
这是在 Spark 2.4 上的 pyspark 中的演示:
>>> from pyspark.sql.functions import *
>>> from pyspark.sql.types import *
>>> df1 = spark.createDataFrame([('a', 9223372034559809871), ('b', 9223372034559809771)], ['group', 'id_int'])
>>> df1=df1.withColumn('id',col('id_int').cast(DecimalType(38,0)))
>>>
>>> df1.show()
+-----+-------------------+-------------------+
|group| id_int| id|
+-----+-------------------+-------------------+
| a|9223372034559809871|9223372034559809871|
| b|9223372034559809771|9223372034559809771|
+-----+-------------------+-------------------+
>>>
>>> df2= spark.createDataFrame([(1, '9223372034559809871'), (2, '9223372034559809771')], ['value', 'id'])
>>> df2.show()
+-----+-------------------+
|value| id|
+-----+-------------------+
| 1|9223372034559809871|
| 2|9223372034559809771|
+-----+-------------------+
>>>
>>> df1.join(df2, ["id"]).show()
+-------------------+-----+-------------------+-----+
| id|group| id_int|value|
+-------------------+-----+-------------------+-----+
|9223372034559809871| a|9223372034559809871| 1|
|9223372034559809871| a|9223372034559809871| 2|
|9223372034559809771| b|9223372034559809771| 1|
|9223372034559809771| b|9223372034559809771| 2|
+-------------------+-----+-------------------+-----+
>>> df1.dtypes
[('group', 'string'), ('id_int', 'bigint'), ('id', 'decimal(38,0)')]
最佳答案
这是因为连接键中的值(非常非常大)而发生的:
我调整了连接条件中的值,它给了我正确的结果:
from pyspark.sql.types import *
df1 = spark.createDataFrame([('a', 9223372034559809871), ('b', 9123372034559809771)],
['group', 'id_int'])
df1=df1.withColumn('id',col('id_int').cast(DecimalType(38,0)))
df2= spark.createDataFrame([(1, '9223372034559809871'), (2, '9123372034559809771')],
['value', 'id'])
df1.join(df2, df1["id"]==df2["id"],"inner").show()
关于apache-spark - 加入 Spark 返回重复的隐式数据类型不匹配,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56502029/