python-3.x - 如何通过替换 pandas 中的 for 循环来减少程序执行时间

Question

我有 12000 个 csv 文件，每个文件有 6000 行。我在我的代码中使用 for 循环，我认为因此我的代码执行时间增加了。如果有人知道如何将这段代码更改为 pandas 包以减少执行时间

for i in range(len(df)):
        if ((df['EOG_Start_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_start_farm'].values[i]-df['EOG_Min_Farm'].values[i])) &((df['EOG_Max_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_Max_Farm'].values[i]-df['EOG_Min_Farm'].values[i]))&((df['Avg'].values[i]>2)):
          #print('EOG')
          df['EOG_flag'].values[i]=1

        if ((df['EOG_Max_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_Max_Farm'].values[i]-df['EOG_Min_Farm'].values[i]))&((df['Avg'].values[i]>2)):
            #print('gust')
            df['Gust_flag'].values[i]=1

注意:这段代码运行良好，只是执行时间较长

Answer 1

您可以使用矢量化解决方案 - 单独创建 bool 掩码，通过 & 链接在一起并在 numpy.where 中设置值:

x = df['EOG_start_farm'].values-df['EOG_Min_Farm'].values
m1 = (df['EOG_Start_model'].values-df['EOG_Min_model'].values) < x
m2 = (df['EOG_Max_model'].values-df['EOG_Min_model'].values) < x
m3 = df['Avg'].values > 2
m23 = m2 & m3

df['EOG_flag'] = np.where(m1 & m2 & m3, 1, df['EOG_flag'].values)
df['Gust_flag'] = np.where(m2 & m3, 1, df['Gust_flag'].values)

性能:

np.random.seed(2019)

N = 6000
c = ['EOG_Start_model','EOG_Min_model','EOG_start_farm','EOG_Min_Farm','EOG_Max_model',
     'EOG_Max_Farm','Avg','EOG_flag','Gust_flag']
df = pd.DataFrame(np.random.rand(N, 9), columns=c)
print (df)

In [91]: %%timeit
    ...: x = df['EOG_start_farm'].values-df['EOG_Min_Farm'].values
    ...: m1 = (df['EOG_Start_model'].values-df['EOG_Min_model'].values) < x
    ...: m2 = (df['EOG_Max_model'].values-df['EOG_Min_model'].values) < x
    ...: m3 = df['Avg'].values > 2
    ...: m23 = m2 & m3
    ...: 
    ...: df['EOG_flag'] = np.where(m1 & m2 & m3, 1, df['EOG_flag'].values)
    ...: df['Gust_flag'] = np.where(m2 & m3, 1, df['Gust_flag'].values)
    ...: 
597 µs ± 6.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [93]: %%timeit
    ...: for i in range(len(df)):
    ...:     if ((df['EOG_Start_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_start_farm'].values[i]-df['EOG_Min_Farm'].values[i])) &((df['EOG_Max_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_Max_Farm'].values[i]-df['EOG_Min_Farm'].values[i]))&((df['Avg'].values[i]>2)):
    ...:           #print('EOG')
    ...:           df['EOG_flag'].values[i]=1
    ...: 
    ...:     if ((df['EOG_Max_model'].values[i]-df['EOG_Min_model'].values[i])<(df['EOG_Max_Farm'].values[i]-df['EOG_Min_Farm'].values[i]))&((df['Avg'].values[i]>2)):
    ...:             #print('gust')
    ...:             df['Gust_flag'].values[i]=1
231 ms ± 1.16 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)