hadoop - 如何读取 pig 中的json数据？

Question

我有以下类型的 json 文件:

{"employees":[
    {"firstName":"John", "lastName":"Doe"},
    {"firstName":"Anna", "lastName":"Smith"},
    {"firstName":"Peter", "lastName":"Jones"}
]}

我正在尝试执行以下 pig 脚本来加载 json 数据

A = load 'pigdemo/employeejson.json' using JsonLoader ('employees:{(firstName:chararray)},{(lastName:chararray)}');

出现错误!!

Unable to recreate exception from backed error: Error: org.codehaus.jackson.JsonParseException: Unexpected end-of-input: expected close marker for ARRAY (from [Source: java.io.ByteArrayInputStream@1553f9b2; line: 1, column: 1]) at [Source: java.io.ByteArrayInputStream@1553f9b2; line: 1, column: 29]

Answer 1

首先，您看到 Unexpected end-of-input 的原因是因为每个重新编码都应该在 1 行中 - 如下所示:

{"employees":[{"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"}]}

现在 - 因为每一行都是员工列表，所以运行下一个命令

A = load '$flurryData' using JsonLoader ('employees:bag {t:tuple(firstName:chararray, lastName:chararray)}');
describe A;
dump A;

给出下一个输出

A: {employees: {t: (firstName: chararray,lastName: chararray)}}

({(John,Doe),(Anna,Smith),(Peter,Jones)})

希望对您有所帮助!