对于每对 src 和 dest 机场城市，我想返回 a 列的百分位数，给定 列的值b。
我可以手动执行此操作:
只有 2 对 src/dest 的示例 df(我的实际 df 中有数千个):
dt src dest a b
0 2016-01-01 YYZ SFO 548.12 279.28
1 2016-01-01 DFW PDX 111.35 -65.50
2 2016-02-01 YYZ SFO 64.84 342.35
3 2016-02-01 DFW PDX 63.81 61.64
4 2016-03-01 YYZ SFO 614.29 262.83
{'a': {0: 548.12,
1: 111.34999999999999,
2: 64.840000000000003,
3: 63.810000000000002,
4: 614.28999999999996,
5: -207.49000000000001,
6: 151.31999999999999,
7: -56.43,
8: 611.37,
9: -296.62,
10: 6417.5699999999997,
11: -376.25999999999999,
12: 465.12,
13: -821.73000000000002,
14: 1270.6700000000001,
15: -1410.0899999999999,
16: 1312.6600000000001,
17: -326.25999999999999,
18: 1683.3699999999999,
19: -24.440000000000001,
20: 583.60000000000002,
21: -5.2400000000000002,
22: 1122.74,
23: 195.21000000000001,
24: 97.040000000000006,
25: 133.94},
'b': {0: 279.27999999999997,
1: -65.5,
2: 342.35000000000002,
3: 61.640000000000001,
4: 262.82999999999998,
5: 115.89,
6: 268.63999999999999,
7: 2.3500000000000001,
8: 91.849999999999994,
9: 62.119999999999997,
10: 778.33000000000004,
11: -142.78,
12: 1675.53,
13: -214.36000000000001,
14: 983.80999999999995,
15: -207.62,
16: 632.13999999999999,
17: -132.53,
18: 422.36000000000001,
19: 13.470000000000001,
20: 642.73000000000002,
21: -144.59999999999999,
22: 213.15000000000001,
23: -50.200000000000003,
24: 338.27999999999997,
25: -129.69},
'dest': {0: 'SFO',
1: 'PDX',
2: 'SFO',
3: 'PDX',
4: 'SFO',
5: 'PDX',
6: 'SFO',
7: 'PDX',
8: 'SFO',
9: 'PDX',
10: 'SFO',
11: 'PDX',
12: 'SFO',
13: 'PDX',
14: 'SFO',
15: 'PDX',
16: 'SFO',
17: 'PDX',
18: 'SFO',
19: 'PDX',
20: 'SFO',
21: 'PDX',
22: 'SFO',
23: 'PDX',
24: 'SFO',
25: 'PDX'},
'dt': {0: Timestamp('2016-01-01 00:00:00'),
1: Timestamp('2016-01-01 00:00:00'),
2: Timestamp('2016-02-01 00:00:00'),
3: Timestamp('2016-02-01 00:00:00'),
4: Timestamp('2016-03-01 00:00:00'),
5: Timestamp('2016-03-01 00:00:00'),
6: Timestamp('2016-04-01 00:00:00'),
7: Timestamp('2016-04-01 00:00:00'),
8: Timestamp('2016-05-01 00:00:00'),
9: Timestamp('2016-05-01 00:00:00'),
10: Timestamp('2016-06-01 00:00:00'),
11: Timestamp('2016-06-01 00:00:00'),
12: Timestamp('2016-07-01 00:00:00'),
13: Timestamp('2016-07-01 00:00:00'),
14: Timestamp('2016-08-01 00:00:00'),
15: Timestamp('2016-08-01 00:00:00'),
16: Timestamp('2016-09-01 00:00:00'),
17: Timestamp('2016-09-01 00:00:00'),
18: Timestamp('2016-10-01 00:00:00'),
19: Timestamp('2016-10-01 00:00:00'),
20: Timestamp('2016-11-01 00:00:00'),
21: Timestamp('2016-11-01 00:00:00'),
22: Timestamp('2016-12-01 00:00:00'),
23: Timestamp('2016-12-01 00:00:00'),
24: Timestamp('2017-01-01 00:00:00'),
25: Timestamp('2017-01-01 00:00:00')},
'src': {0: 'YYZ',
1: 'DFW',
2: 'YYZ',
3: 'DFW',
4: 'YYZ',
5: 'DFW',
6: 'YYZ',
7: 'DFW',
8: 'YYZ',
9: 'DFW',
10: 'YYZ',
11: 'DFW',
12: 'YYZ',
13: 'DFW',
14: 'YYZ',
15: 'DFW',
16: 'YYZ',
17: 'DFW',
18: 'YYZ',
19: 'DFW',
20: 'YYZ',
21: 'DFW',
22: 'YYZ',
23: 'DFW',
24: 'YYZ',
25: 'DFW'}}
我想要每组 src 和 dest 对的百分位数。所以每对应该只有 1 个百分位值。我只想对每个 src 和 dest 执行给定 b 的百分位数，其中 date = 2017-01-01对每一对的整个列 a 进行配对。有道理吗？
例如，我可以针对特定的一对 手动执行此操作，即src=YYZ 和 dest=SFT:
from scipy import stats
import datetime as dt
import pandas as pd
p0 = dt.datetime(2017,1,1)
# lets slice df for src=YYZ and dest = SFO
x = df[(df.src =='YYZ') &
(df.dest =='SFO') &
(df.dt ==p0)].b.values[0]
# given B, what percentile does it fall in for the entire column A for YYZ, SFO
stats.percentileofscore(df['a'],x)
61.53846153846154
在上述情况下，我为 YYZ 和 SFO 对手动执行了此操作。但是，我的 df 中有数千对。
如何使用 pandas 功能 对其进行矢量化，而不是遍历每一对？
一定有办法在一个函数上使用 groupby 和使用 apply 吗？
我想要的 df 应该是这样的:
src dest percentile
0 YYZ SFO 61.54
1 DFW PDX 23.07
2 XXX YYY blahblah1
3 AAA BBB blahblah2
...
更新:
我实现了以下内容:
def b_percentile_a(df,x,y,b):
z = df[(df['src'] == x ) & (df['dest'] == y)].a
r = stats.percentileofscore(z,b)
return r
b_vector_df = df[df.dt == p0]
b_vector_df['p0_a_percentile_b'] = \
b_vector_df.apply(lambda x: b_percentile_a(df,x.src,x.dest,x.b), axis=1)
100 对需要 5.16 秒。我有 55,000 对。所以这将花费 ~50 分钟。我需要运行这个 36 次，所以它需要 几天 的运行时间。
一定有更快的方法吗？
最佳答案
节省了令人难以置信的时间!
输出:
a_list 的大小:49998 个随机唯一值
percentile_1(您给定的 df - scipy)
计算百分位数 104 次 - 0:00:07.777022 中的 104 条记录
percentile_9(类 PercentileOfScore(rank_searchsorted_list) 使用给定的 df)
计算百分位数 104 次 - 0:00:00.000609 中有 104 条记录
_ dt src dest a b pct scipy _
0: 2016-01-01 YYZ SFO 54812 279.28 74.81299251970079 74.8129925197
1: 2016-01-01 DFW PDX 111.35 -65.5 24.66698667946718 24.6669866795
2: 2016-02-01 YYZ SFO 64.84 342.35 76.4810592423697 76.4810592424
3: 2016-02-01 DFW PDX 63.81 61.64 63.84655386215449 63.8465538622
...
24: 2017-01-01 YYZ SFO 97.04 338.28 76.3570542821712 76.3570542822
25: 2017-01-01 DFW PDX 133.94 -129.69 21.4668586743469 21.4668586743
查看 scipy.percentileofscore 的实现，我发现整个 list( a )
在 percentileofscore 的每次调用中被复制、插入、排序、搜索。
我实现了自己的 class PercentileOfScore
import numpy as np
class PercentileOfScore(object):
def __init__(self, aList):
self.a = np.array( aList )
self.a.sort()
self.n = float(len(self.a))
self.pct = self.__rank_searchsorted_list
# end def __init__
def __rank_searchsorted_list(self, score_list):
adx = np.searchsorted(self.a, score_list, side='right')
pct = []
for idx in adx:
# Python 2.x needs explicit type casting float(int)
pct.append( (float(idx) / self.n) * 100.0 )
return pct
# end def _rank_searchsorted_list
# end class PercentileOfScore
我认为 def percentile_7 不能满足您的需求。 dt 将不予考虑。
PctOS = None
def percentile_7(df_flat):
global PctOS
result = {}
for k in df_flat.pair_dict.keys():
# df_flat.pair_dict = { 'src.dst': [b,b,...bn] }
result[k] = PctOS.pct( df_flat.pair_dict[k] )
return result
# end def percentile_7
在您的手动示例中，您使用了整个 df.a。在此示例中，它的 dt_flat.a_list，但我不确定这是否是您想要的？
from PercentileData import DF_flat
def main():
# DF_flat.data = {'dt.src.dest':[a,b]}
df_flat = DF_flat()
# Instantiate Global PctOS
global PctOS
# df_flat.a_list = [a,a,...an]
PctOS = PercentileOfScore(df_flat.a_list)
result = percentile_7(df_flat)
# result = dict{'src.dst':[pct,pct...pctn]}
使用 Python 测试:3.4.2 和 2.7.9 - numpy:1.8.2
关于python - 向量化 A 列 B 列的百分位数值(对于组)，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/42076126/