目标 = 通过 dask 数据框进行多列分组,并过滤掉包含少于 3 行的组。
基于这篇文章: Filtering grouped df in Dask
我能够计算每个 groupby 对象的大小,但我不知道如何将它从多列 groupby 映射回我的数据框。我尝试了以下多种变体但无济于事:
a = input_df.groupby(["FeatureID", "region"])["Target"].size()
s = input_df[["FeatureID", "region"]].map(a)
它对单列 groupby 非常有效。
解决方案
感谢@jezrael,我能够想出以下解决方案:
a = input_df.groupby(["FeatureID", "region"])["Target"].nunique().to_frame("feature_div")
input_df = input_df.join(a, on=["FeatureID", "region"])
# filter out features below diversity threshold
diversified = input_df[input_df.feature_div >= diversity_threshold]
最佳答案
您需要使用to_frame
join
:
a = input_df.groupby(["FeatureID", "region"])["Target"].size().to_frame('New')
input_df = input_df.join(a, on=["FeatureID", "region"])
示例:
import pandas as pd
from dask import dataframe as dd
input_df = pd.DataFrame({
'FeatureID':[4,5,4,5,5,4],
'region':list('aaabbb'),
'Target':[7,8,9,4,2,3],
})
print (input_df)
FeatureID region Target
0 4 a 7
1 5 a 8
2 4 a 9
3 5 b 4
4 5 b 2
5 4 b 3
sd = dd.from_pandas(input_df, npartitions=3)
print (sd)
FeatureID region Target
npartitions=3
0 int64 object int64
2 ... ... ...
4 ... ... ...
5 ... ... ...
Dask Name: from_pandas, 3 tasks
a = sd.groupby(["FeatureID", "region"])["Target"].size().to_frame('New')
out = sd.join(a, on=["FeatureID", "region"]).compute()
print (out)
FeatureID region Target New
0 4 a 7 2
1 5 a 8 1
2 4 a 9 2
3 5 b 4 2
4 5 b 2 2
5 4 b 3 1
关于python - 多列 Groupby 大小的 Dask 过滤器数据框,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54682524/