hadoop - 从 pig 的生产线中提取

Question

我正在尝试按网址对数据进行分组。我的数据当前存储在一行中。例如。:
{“mobile”，“country:US”，“url:1234.com”，“newuser:y”}等。

这是我到目前为止的内容:

RAW = LOAD '/data/events/raw/2014-08-21/' as (line:chararray);
A = FILTER RAW BY (INDEXOF(line,'mobile') != -1)
B = LIMIT A 800;
URL = GROUP B BY (INDEXOF(line, 'url'));
STORE URL INTO '/user/hadoopuser/RS_traffic.txt';

我如何从字符串中提取URL以便对其进行分组？我可以使用正则表达式吗？

Answer 1

您可以使用REGEX_EXTRACT()函数:

REGEX_EXTRACT Javadoc

RAW = LOAD '/data/events/*' AS (line:chararray);
C = FOREACH RAW GENERATE REGEX_EXTRACT(value, '<your_pattern>', 1) AS url:chararray;
A = FILTER RAW BY (INDEXOF(line,'mobile') != -1)
URL = GROUP C BY url;
....
STORE URL INTO '/user/hadoopuser/RS_traffic.txt';