我经常需要通过df[df['col_name']=='string_value']
过滤pandas dataframe df
,我想加快行选择操作,有没有快速的方法呢?
例如,
In [1]: df = mul_df(3000,2000,3).reset_index()
In [2]: timeit df[df['STK_ID']=='A0003']
1 loops, best of 3: 1.52 s per loop
1.52s可以缩短吗?
注意:
mul_df()
是创建多级数据框的函数:
>>> mul_df(4,2,3)
COL000 COL001 COL002
STK_ID RPT_Date
A0000 B000 0.6399 0.0062 1.0022
B001 -0.2881 -2.0604 1.2481
A0001 B000 0.7070 -0.9539 -0.5268
B001 0.8860 -0.5367 -2.4492
A0002 B000 -2.4738 0.9529 -0.9789
B001 0.1392 -1.0931 -0.2077
A0003 B000 -1.1377 0.5455 -0.2290
B001 1.0083 0.2746 -0.3934
下面是mul_df()的代码:
import itertools
import numpy as np
import pandas as pd
def mul_df(level1_rownum, level2_rownum, col_num, data_ty='float32'):
''' create multilevel dataframe, for example: mul_df(4,2,6)'''
index_name = ['STK_ID','RPT_Date']
col_name = ['COL'+str(x).zfill(3) for x in range(col_num)]
first_level_dt = [['A'+str(x).zfill(4)]*level2_rownum for x in range(level1_rownum)]
first_level_dt = list(itertools.chain(*first_level_dt)) #flatten the list
second_level_dt = ['B'+str(x).zfill(3) for x in range(level2_rownum)]*level1_rownum
dt = pd.DataFrame(np.random.randn(level1_rownum*level2_rownum, col_num), columns=col_name, dtype = data_ty)
dt[index_name[0]] = first_level_dt
dt[index_name[1]] = second_level_dt
rst = dt.set_index(index_name, drop=True, inplace=False)
return rst
最佳答案
我一直想为 DataFrame 对象添加二进制搜索索引。您可以采用按列排序的 DIY 方法并自己执行此操作:
In [11]: df = df.sort('STK_ID') # skip this if you're sure it's sorted
In [12]: df['STK_ID'].searchsorted('A0003', 'left')
Out[12]: 6000
In [13]: df['STK_ID'].searchsorted('A0003', 'right')
Out[13]: 8000
In [14]: timeit df[6000:8000]
10000 loops, best of 3: 134 µs per loop
这很快,因为它总是检索 View 并且不复制任何数据。
关于python - 如何通过字符串匹配加速 Pandas 行过滤?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/16384332/