python - 如何使用 groupby 执行引用数据框中数据子集上一行的函数

Question

我有一些日志数据代表一个项目 (id) 和一个操作开始的时间戳，我想确定每个项目上的操作之间的时间。

例如，我有一些如下所示的数据:

data = [{"timestamp":"2019-05-21T14:17:29.265Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T14:21:49.722Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.695Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.696Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-22T07:51:17.49Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T08:11:13.948Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:52:59.897Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.406Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.481Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-21T14:23:08.147Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:29:18.228Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.831Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.834Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:02:19.072Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:02:34.867Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:12:28.877Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.567Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.582Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T09:58:02.185Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"},{"timestamp":"2019-05-21T10:07:24.044Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"}]
stack = pd.DataFrame(data)
stack.head()

我已经尝试获取所有唯一 ID 来拆分数据框，然后获取索引所花费的时间与原始集合重新组合，但该函数在大型数据集上非常慢并且弄乱了两个索引 和时间戳顺序导致结果未匹配。

import ciso8601 as time
records = []
for i in list(stack.id.unique()):
    dff = stack[stack.id == i]
    time_taken = []
    times = []
    i = 0
    for _, row in dff.iterrows():
        if bool(times):
            print(_)
            current_time = time.parse_datetime(row.timestamp)
            prev_time = times[i]
            time_taken = current_time - prev_time
            times.append(current_time)
            i+=1
            records.append(dict(index = _, time_taken = time_taken.seconds))
        else:
            records.append(dict(index = _, time_taken = 0))
            times.append(time.parse_datetime(row.timestamp))

x = pd.DataFrame(records).set_index('index')
stack.merge(x, left_index=True, right_index=True, how='inner')

是否有一种简洁的 pandas groupby 和 apply 方法来执行此操作，这样我就不必拆分帧并将其存储在内存中以便可以引用子集中的前一行？

谢谢

Answer 1

您可以使用 GroupBy.diff :

stack['timestamp'] = pd.to_datetime(stack['timestamp'])
stack['timestamp']= (stack.sort_values(['id','timestamp'])
                            .groupby('id')
                            .diff()['timestamp']
                            .dt.total_seconds()
                            .round().fillna(0))

print(stack['time_taken'])
0         0.0
1       260.0
2      3276.0
3         0.0
4         0.0
5      1196.0
6     13306.0
7         4.0
8         0.0
9         0.0
10      370.0
11     2872.0
...

---

如果您希望生成的数据框按日期排序，请执行以下操作:

stack['timestamp'] = pd.to_datetime(stack['timestamp']) 
stack = stack.sort_values(['id','timestamp'])
stack['time_taken'] = (stack.groupby('id')
                            .diff()['timestamp'] 
                            .dt.total_seconds() 
                            .round()
                            .fillna(0))