python - 如何在pig或hive中使用array_agg()聚合函数

Question

我有以下数据:

================================================================
session_id                       screen_name  screen_launch_time
================================================================
990004916946605-1404157897784    screen1      1404157898275
990004916946605-1404157897784    screen2      1404157898337
990004947764274-1435162269418    screen1      1435162274044
990004947764274-1435162269418    screen3      1435162274081

我想使用 array_agg 函数来获取以下格式的数据:

=========================================================
session_id                      screen_flow         count
=========================================================
990004916946605-1404157897784   screen1->screen2    1
990004947764274-1435162269418   screen1->screen3    1

有人尝试过编写 UDAF 或 python 脚本来实现 array_agg 函数中使用的逻辑吗？

请分享您的想法。

Answer 1

只需按 session_id 进行分组，连接 screen_name，然后计算每组的记录数。如果您不想构建 brickhouse jar 中，您可以使用 collect_list() 而不是 collect() (但我不推荐它)。

查询:

add jar /path/to/jars/brickhouse-0.7.1.jar;
create temporary function collect as "brickhouse.udf.collect.CollectUDAF";

select session_id, screen_flow
  , count(*) count
from (
  select session_id
    , concat_ws('->', collect(screen_name)) screen_flow
  from db.table
  group by session_id ) x
group by session_id, screen_flow

输出:

990004916946605-1404157897784   screen1->screen2    1
990004947764274-1435162269418   screen1->screen3    1