python - 在 groupby 对象内的列列表上生成滚动计算的更快方法

Question

我创建了这个函数来计算 df 中一系列专长的滚动统计数据。该函数按预期工作，但在我的 df 上运行大约需要 30 分钟，该 df 大约有 100 万行。在 python/pandas 中是否有更快的方法来做到这一点？

def add_rolling_vars(df, feats, amounts, group):
#creates rolling stats for a list of feats(columns) over a list of amounts[12,48](window sizes)
#grouped by a group like $gvkey or $sector
orig_feats = feats.copy()
new_feats= []
for amount in amounts:
    for name in feats:
        df[group+'_'+name+f'_{amount}_sma'] = df.groupby(group)[name].rolling(amount,1).mean().values
        df[group+'_'+name+f'_{amount}_std'] = df.groupby(group)[name].rolling(amount,1).std().values
        df[group+'_'+name+f'_{amount}_min'] = df.groupby(group)[name].rolling(amount,1).min().values
        df[group+'_'+name+f'_{amount}_max'] = df.groupby(group)[name].rolling(amount,1).max().values
        df[group+'_'+name+f'_{amount}_med'] = df.groupby(group)[name].rolling(amount,1).median().values
        df[group+'_'+name+f'_{amount}_25Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.25).values
        df[group+'_'+name+f'_{amount}_75Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.75).values

作为示例，我还创建了此函数，该函数在同一数据集上运行大约 1 分钟。显然，它是不同的，因为它不必遍历行窗口，但我仍然可以传递一个壮举列表而不是壮举中的名称，然后使用列表理解命名方案将整个转换后的输出添加到我的数据帧中:

def add_cat_stats(df,feats,group):
    #feats is a list of continuous feats to compute the monthly stats of       
    df[[group+'_'+name+'_avg' for name in feats]] = df.groupby([group,'Date'])[feats].transform('mean')
    df[[group+'_'+name+'_std' for name in feats]] = df.groupby([group,'Date'])[feats].transform('std')
    df[[group+'_'+name+'_min' for name in feats]] = df.groupby([group,'Date'])[feats].transform('min')
    df[[group+'_'+name+'_max' for name in feats]] = df.groupby([group,'Date'])[feats].transform('max')
    df[[group+'_'+name+'_med' for name in feats]] = df.groupby([group,'Date'])[feats].transform('median')

更新

len(数量) = 2

长度(功绩)= 16

Answer 1

我无法让@John Zwinck 代码工作，但它确实给了我重新格式化代码的想法，这将时间从 30 分钟减少到 4 分 45 秒，这太棒了!进一步降低它会很好，但这是一个可行的解决方案:

def add_rolling_vars(df, feats, amounts, group):
    for amount in amounts:
        grouped = df.groupby(group)[feats].rolling(amount,1)
        prefix = ['_'.join([group, name, str(amount)]) for name in feats]
        df[[pre+'_sma' for pre in prefix]] = grouped.mean().reset_index(0,drop=True)
        df[[pre+'_std' for pre in prefix]] = grouped.std().reset_index(0,drop=True)
        df[[pre+'_min' for pre in prefix]] = grouped.min().reset_index(0,drop=True)
        df[[pre+'_max' for pre in prefix]] = grouped.max().reset_index(0,drop=True)
        df[[pre+'_med' for pre in prefix]] = grouped.median().reset_index(0,drop=True)
        df[[pre+'_25Q' for pre in prefix]] = grouped.quantile(.25).reset_index(0,drop=True)
        df[[pre+'_75Q' for pre in prefix]] = grouped.quantile(.75).reset_index(0,drop=True)