Python - Pandas - 如何在数据帧合并后从 to_json 中删除空值

Question

我正在构建一个过程来“外部连接”两个 csv 文件并将结果导出为 json 对象。

# read the source csv files
firstcsv = pandas.read_csv('file1.csv',  names = ['main_index','attr_one','attr_two'])
secondcsv = pandas.read_csv('file2.csv',  names = ['main_index','attr_three','attr_four'])

# merge them
output = firstcsv.merge(secondcsv, on='main_index', how='outer')

jsonresult = output.to_json(orient='records')
print(jsonresult)

现在，两个 csv 文件是这样的:

file1.csv:
1, aurelion, sol
2, lee, sin
3, cute, teemo

file2.csv:
1, midlane, mage
2, jungler, melee

我希望输出结果 json:

[{"main_index":1,"attr_one":"aurelion","attr_two":"sol","attr_three":"midlane","attr_four":"mage"},
{"main_index":2,"attr_one":"lee","attr_two":"sin","attr_three":"jungler","attr_four":"melee"},
{"main_index":3,"attr_one":"cute","attr_two":"teemo"}]

相反，我使用 main_index = 3 上线

{"main_index":3,"attr_one":"cute","attr_two":"teemo","attr_three":null,"attr_four":null}]

因此空值会自动添加到输出中。 我想删除它们 - 我环顾四周，但找不到合适的方法。

希望有人能帮助我!

Answer 1

由于我们使用的是 DataFrame，pandas 会用 NaN 来“填充”值，即

>>> print(output)
      main_index   attr_one attr_two attr_three attr_four
0           1   aurelion      sol    midlane      mage
1           2        lee      sin    jungler     melee
2           3       cute    teemo        NaN       NaN

我在 pandas.to_json 文档中看不到任何跳过空值的选项:https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_json.html

所以我想出的方法是重新构建 JSON 字符串。对于数百万行的大型数据集来说，这可能性能不太好(但联盟中的冠军不到 200 个，所以不应该是一个大问题!)

from collections import OrderedDict
import json

jsonresult = output.to_json(orient='records')
# read the json string to get a list of dictionaries
rows = json.loads(jsonresult)

# new_rows = [
#     # rebuild the dictionary for each row, only including non-null values
#     {key: val for key, val in row.items() if pandas.notnull(val)}
#     for row in rows
# ]

# to maintain order use Ordered Dict
new_rows = [
    OrderedDict([
        (key, row[key]) for key in output.columns
        if (key in row) and pandas.notnull(row[key])
    ])
   for row in rows
]

new_json_output = json.dumps(new_rows)

你会发现new_json_output已经删除了所有具有NaN值的键，并保持了顺序:

>>> print(new_json_output)
[{"main_index": 1, "attr_one": " aurelion", "attr_two": " sol", "attr_three": " midlane", "attr_four": " mage"},
 {"main_index": 2, "attr_one": " lee", "attr_two": " sin", "attr_three": " jungler", "attr_four": " melee"},
 {"main_index": 3, "attr_one": " cute", "attr_two": " teemo"}]