python - 如何分解 Pandas 中的嵌套 json 结构？

Question

我正在处理一个文件，每行包含一个 json block 。每行看起来像这样:

{"a":3,"b":10,"unnecessaryList":[{"value":12,"colName":"c"},{"value":792,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561653"}

json 的生成者选择了不必要的嵌套结构，而平面结构就足够了。也就是说，我想以更明显的扁平结构将数据读入 Pandas DataFrame，其中包含列“a”、“b”、“c”、“d”、“e”、“index”。到目前为止，我想到的最好的办法是以不同的方式处理文件两次:

import pandas as pd
from pandas.io.json import json_normalize, loads

raw_json = pd.read_json('sample.json', lines=True)
raw_json.set_index('index', inplace=True)

with open('sample.json') as f:
    lines = f.readlines()
    exploded_columns = pd.concat([json_normalize(loads(l), 'unnecessaryList', 'index').pivot(index='index', columns='colName', values='value') for l in lines])

data = pd.merge(raw_json[['a', 'b']], exploded_columns, left_index=True, right_index=True)

有没有办法避免这样两次读取数据？ Pandas 是否提供了一些可以避免我想出的 concat/normalize/pivot/merge 垃圾的功能？

Answer 1

您可以使用:

df = pd.read_json('sample.json', lines=True)
#create Multiindex from 3 columns and select unnecessaryList for Series
s = df.set_index(['a','b','index'])['unnecessaryList']
print (s)
a  b   index      
3  10  -1417561653    [{'value': 12, 'colName': 'c'}, {'value': 792,...
       -1417561655    [{'value': 13, 'colName': 'c'}, {'value': 794,...
       -1417561658    [{'value': 14, 'colName': 'c'}, {'value': 795,...
Name: unnecessaryList, dtype: object

#create DataFrame for each dict and concat, transpose
L = [pd.DataFrame(x).set_index('colName')['value'] for x in s]
df = (pd.concat(L, axis=1, keys=s.index)
       .T
       .reset_index(level=[0,1])
       .rename(columns={'level_0':'a','level_1':'b'})
       .rename_axis(None, 1))
print (df)
             a   b   c    d    e
-1417561653  3  10  12  792  645
-1417561655  3  10  13  794  645
-1417561658  3  10  14  795  645

输入json数据:

{"a":3,"b":10,"unnecessaryList":[{"value":12,"colName":"c"},{"value":792,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561653"}
{"a":3,"b":10,"unnecessaryList":[{"value":13,"colName":"c"},{"value":794,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561655"}
{"a":3,"b":10,"unnecessaryList":[{"value":14,"colName":"c"},{"value":795,"colName":"d"},{"value":645,"colName":"e"}],"index":"-1417561658"}