python - 合并具有许多稀疏列的两个 Pandas DataFrame 会导致 DataFrame 需要不成比例的大量内存

Question

当合并两个稀疏数据帧时，结果数据帧在内存中变得不成比例。我想知道为什么会这样。对新数据帧的操作非常缓慢。我尝试了不同的方法来减少内存占用，但没有奏效。例如，使用不同的 fill_values(0 或 0.0)，在密集列和稀疏列之间来回转换，重置索引，删除指标列，制作合并数据帧的副本。
任何想法是什么导致了这个问题以及如何解决它？我正在使用 Pandas 1.1.1 版。
以下是有关数据帧的一些信息:
DF1:

Int64Index: 113774 entries, 0 to 113773  
Columns: 24155 entries  
dtypes: Sparse[float32, 0](1), Sparse[float64, 0](24149), float32(2), int32(2), int8(1)  
memory usage: 7.3 MB  

DF2:

Int64Index: 128507 entries, 0 to 128506  
Columns: 1962 entries  
dtypes: Sparse[float64, 0](1957), float32(1), int16(1), int32(2), int8(1)  
memory usage: 10.0 MB  

合并 DF:

Int64Index: 136333 entries, 0 to 136332  
Columns: 26115 entries  
dtypes: Sparse[float64, 0](26107), category(1), float32(4), int32(2), int8(1)  
memory usage: 6.3 GB  

这就是我构建新数据框的方式:

df_joined= df1.merge(
    df2, 
    on=key_cols, 
    how='outer', 
    indicator='df_indicator', 
    suffixes=['_DF1', '_DF2']
)

# replace null values
null_cols = pp.get_null_columns(df_joined)
for field in null_cols:
    df_joined[field]= df_joined[field].fillna(0.0)

Answer 1

我对这个问题进行了一些调查，但仍然觉得很奇怪。所以这并不意味着作为答案，而只是为了展示 Pandas 奇怪行为的一部分。
看起来问题是，如果 fillna用于填写设置为fill_value的值的稀疏数组，它不被识别为 fill_value因此存储为常规浮点数。请看下面的代码。只有在重新组织稀疏列之后，空间才会被释放。

import pandas as pd
import numpy as np

def make_sparse(df):
    for col in df.columns[1:]:
        # make sure we have plenty zeros in the dataframe columns
        # by setting everything lower than 0 and all greather than 0.2
        # to zero
        # skip the first column
        indexer= df[col].between(0.0, 0.2)
        df.loc[~indexer, col]= 0.0
        df[col]= pd.arrays.SparseArray(df[col], fill_value=0.0)
    return df

# build a dataframe with one regular float column and two sparse columns
df1= pd.DataFrame(np.random.randn(10000, 3), columns=['num1', 'num2', 'num3'])
df1= make_sparse(df1)
df1['id']= list(range(df1.shape[0]))
df1.set_index('id', inplace=True)

df1.memory_usage()
"""
This results in the following output in my example
num1     80000
num2      9408
num3      9432
You can see, that num1 and num2 need much less space due to the sparse structure
"""

# create another dataframe with one regular and one sparse column
# the index is the same as for df1, but we only have one row for
# each other record in df1 (the ones with an even id
df2= pd.DataFrame(np.random.randn(5000, 2), columns=['num_a', 'num_b'])
df2= make_sparse(df2)
df2['id']= list(range(0, df2.shape[0]*2, 2))
df2.set_index('id', inplace=True)

df2.memory_usage()
"""
Index    40000
num_a    40000
num_b     5124
"""

# now merge the two
df3= df1.merge(df2, left_index=True, right_index=True, how='left')

df3.memory_usage()
"""
We still have the same memory usage of the columns from df1
also the column num_a is ok, since we have it now for each row
in df1, but num_b needs much more space as before, because it contains
Na-Values
Index    407680
num1      80000
num2       9408
num3       9432
num_a     80000
num_b     65124
"""

# so let's try replacing the Na-values by 0.0
df4['num_b']= df4['num_b'].fillna(0.0)
df3.memory_usage()
"""
This doesn't change anything.
It looks like the 0.0 inserted for the Nas are just
treated as any other float values and not like the fill_value
of the sparse array:
Index    407680
num1      80000
num2       9408
num3       9432
num_a     80000
num_b     65124
"""

# now rebuild the sparse column num_b
df4= df3.copy()
df4['num_b']= pd.arrays.SparseArray(df4['num_b'], fill_value=0.0)
df4.memory_usage()
"""
Now num_b has the same memeory usage as in df2
Index    407680
num1      80000
num2       9408
num3       9432
num_a     80000
num_b      5124
"""

这种行为有原因吗？
这可能是 Pandas 中的错误吗？