hadoop - 从 PIG 包中提取元组

Question

文件内容(test.txt):

Some    specific    column      value: x192.168.1.2     blah       blah
Some    specific    row        value: y192.168.1.3      blah       blah
Some    specific    field      value: z192.168.1.4     blah      blah

pig 查询:

A = LOAD 'test.txt' USING PigStorage('\t') AS (data1: chararray , data2: chararray , data3: chararray, data4: chararray , data5: chararray , data6: chararray);

B = foreach A generate data3, data4;

C = filter B by data3 matches 'row';

D = foreach C generate data4;

E = foreach D generate TOKENIZE(data4);

输出:

((value:), (y192.168.1.3))

现在我想在这个输出包中提取特定的元组，比如第二个元组 (y192.168.1.3)。 在此之后我想提取IP地址。我正在尝试使用 UDF，但卡住了。

Answer 1

这是我会做的。

pig 脚本

A = LOAD 'test.txt' USING PigStorage('\t') AS (data1: chararray , data2: chararray , data3: chararray, data4: chararray , data5: chararray , data6: chararray);
B = foreach A generate data3, data4;
C = filter B by data3 matches 'row';
D = foreach C generate data4;
E = foreach D generate REGEX_EXTRACT($0,'value: .([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+).*', 1);

输出

(192.168.1.3)

如果需要，您可以使用更疯狂的正则表达式来提取 IP 地址:Extract ip addresses from Strings using regex