python - 将 JSON 键值分解为新行

Question

我正在寻找转换嵌套 JSON 并将其分解为新行。

当前数据:

+-----------+-----------+-----------------------------+
|         id|      site |                         sale|
+-----------------------------------------------------+
|        abc|          6|{"2019-05-08T00:00:00": null, "2019-05-09T00:00:00": [{"key1": 2, "key2": 0}]}         |
|        def|          5|{"2019-05-08T00:00:00": [{"key1": 22, "key2": 10}], "2019-05-09T00:00:00": null}       |
|        ghi|          4|{"2019-05-08T00:00:00": null, "2019-05-09T00:00:00": null}         |

期望的结果

+-----------+-----------+------------------------+-----------+-----------+
|         id|      site |                saledate|       key1|       key2|
+-------------------------------------------------------------------------
|        abc|          6|     2019-05-08T00:00:00|       null|       null|
|        abc|          6|     2019-05-09T00:00:00|          2|          0|
|        def|          5|     2019-05-08T00:00:00|         22|         10|
|        def|          5|     2019-05-09T00:00:00|       null|       null|
|        ghi|          4|     2019-05-08T00:00:00|       null|       null|
|        ghi|          4|     2019-05-09T00:00:00|       null|       null|

我尝试过的:

• explode 仅适用于 Array 类型。

new_df = old_df.withColumn('saledate', explode('sale'))

• 我可以通过执行以下操作来获取每个日期的属性。但是，我必须单独指定每个日期，这并不理想。

new_df = old_df
  .withColumn('sale_collection', explode('sale.2019-05-08T00:00:00'))
  .withColumn('key1', col('sale_collection').getItem('key1')
  .withColumn('key2', col('sale_collection').getItem('key2')

Answer 1

您可以使用from_json将字符串转换为 Map 然后爆炸:

from pyspark.sql.functions import from_json

df.withColumn('sale', from_json('sale', 'map<string,array<struct<key1:int,key2:int>>>')) \
  .selectExpr('*', 'explode_outer(sale) as (saledate, keys)') \
  .selectExpr('id', 'site', 'saledate', 'inline_outer(keys)') \
  .show()
+---+----+-------------------+----+----+
| id|site|           saledate|key1|key2|
+---+----+-------------------+----+----+
|abc|   6|2019-05-08T00:00:00|null|null|
|abc|   6|2019-05-09T00:00:00|   2|   0|
|def|   5|2019-05-08T00:00:00|  22|  10|
|def|   5|2019-05-09T00:00:00|null|null|
|ghi|   4|2019-05-08T00:00:00|null|null|
|ghi|   4|2019-05-09T00:00:00|null|null|
+---+----+-------------------+----+----+

注意: inline_outer和 inline是 SparkSQL 内置函数，用于分解结构数组。