python - pandas:合并多个数据框

Question

我有几个数据帧 df1、df2...，其中包含重复的数据、部分重叠的列和行((见下文)

如何将所有数据帧集中到一个数据帧中。

df1 = pd.DataFrame({'A': [1,2], 'B': [4,5]}, index=['a', 'b'])

df2 = pd.DataFrame({'B': [5,6], 'C': [8,9]}, index=['b', 'c'])

df3 = pd.DataFrame({'A': [2,3], 'B': [5,6]}, index=['b', 'c'])

df4 = pd.DataFrame({'C': [7,8], index=['a', 'b'])

df5 = pd.DataFrame({'A': [1], 'B': [4], 'C': [7]}, index=['a'])

....

添加:示例数据结构

   A   B   C
a  1   4   7
b  2   5   8
c  3   6   9

补充:我真正寻找的是以下脚本的更有效方法，这对于大数据帧来说确实很慢

dfs =[df1, df2, df3, df4, df5]
cols, rows = [], []
for df in dfs:
    cols = cols + df.columns.tolist()
    rows = rows + df.index.tolist()
cols = np.unique(cols)
rows = np.unique(rows)

merged_dfs = pd.DataFrame(data=np.nan, columns=cols, index=rows)

for df in dfs:
    for col in df.columns:
        for row in df.index:
            merged_dfs[col][row] = df[col][row]

快速简单的解决方案(2015 年 23 月 23 日添加)

dfs =[df1, df2, df3, df4, df5]

# create empty DataFrame with all cols and rows
cols, rows = [], []
for df_i in dfs:
    cols = cols + df_i.columns.tolist()
    rows = rows + df_i.index.tolist()
cols = np.unique(cols)
rows = np.unique(rows)       
df = pd.DataFrame(data=np.NaN, columns=cols, index=rows) 

# fill DataFrame
for df_i in dfs:
    df.loc[df_i.index, df_i.columns] = df_i.values

Answer 1

具有索引保存

这是保留索引的更新版本:

from functools import reduce

dfs = [df1, df2, df3, df3, df5]

def my_merge(df1, df2):
    res = pd.merge(df1, df2, how='outer', left_index=True, right_index=True)
    cols = sorted(res.columns)
    pairs = []
    for col1, col2 in zip(cols[:-1], cols[1:]):
        if col1.endswith('_x') and col2.endswith('_y'):
            pairs.append((col1, col2))
    for col1, col2 in pairs:
        res[col1[:-2]] = res[col1].combine_first(res[col2])
        res = res.drop([col1, col2], axis=1)
    return res

print(reduce(my_merge, dfs))

输出:

   A  B  C
a  1  4  7
b  2  5  8
c  3  6  9

不保留索引

这是一种方法:

from functools import reduce  # Python 3 only

dfs = [df1, df2, df3, df3, df5]

def my_merge(df1, df2):
    return pd.merge(df1, df2, how='outer')

merged_dfs = reduce(my_merge, dfs)

结果:

   A  B   C
0   1  4 NaN
1   2  5   8
2 NaN  6   9
3   3  6 NaN
4   1  4   7

您可以通过设置方式来调整连接方法:

how : {'left', 'right', 'outer', 'inner'}, default 'inner'

• left: use only keys from left frame (SQL: left outer join)
• right: use only keys from right frame (SQL: right outer join)
• outer: use union of keys from both frames (SQL: full outer join)
• inner: use intersection of keys from both frames (SQL: inner join)

如果您喜欢 lambda，请使用此版本以获得相同的结果:

reduce(lambda df1, df2: pd.merge(df1, df2, how='outer'), dfs)