我是Pig脚本的新手。请帮我解决这个问题。
我不知道我要去哪里错了。
我的资料
(catA,myid_1,2014,store1,appl)
(catA,myid_2,2014,store1,milk)
(catA,myid_3,2014,store1,appl)
(catA,myid_4,2014,store1,milk)
(catA,myid_5,2015,store1,milk)
(catB,myid_6,2014,store2,milk)
(catB,myid_7,2014,store2,appl)
以下是预期的结果
(catA,2014,milk,2)
(catA,2014,apple,2)
(catA,2015,milk,1)
(catB,2014,milk,1)
(catB,2014,apple,1)
需要根据类别,年份计算食物的数量。
以下是我的 pig 脚本
list = LOAD 'shop' USING PigStorage(',') AS (category:chararray,id:chararray,mdate:chararray,my_store:chararray,item:chararray);
list_of = FOREACH list GENERATE category,SUBSTRING(mdate,0,4) as my_date,my_store,item;
StoreG = GROUP list_of BY (category,my_date,my_store);
result = FOREACH StoreG
{
food_list = FOREACH list_of GENERATE item;
food_count = DISTINCT food_list;
GENERATE FLATTEN(group) AS (category,my_date,my_store),COUNT(food_count);
}
DUMP result;
我上面脚本的输出如下
(catA,2014,store1,2)
(catA,2015,store1,1)
(catB,2014,store2,2)
任何人都可以让我知道我在脚本中哪里写错了
谢谢
最佳答案
一种方法,不是最优雅但可行的示例:
list = LOAD 'shop' USING PigStorage(',') AS (category:chararray,id:chararray,mdate:chararray,my_store:chararray,item:chararray);
list_of = FOREACH list GENERATE category,SUBSTRING(mdate,0,4) AS my_date,my_store,item;
StoreG = GROUP list_of BY (category,my_date,my_store,item);
result = FOREACH StoreG GENERATE
group.category AS category,
group.my_date AS my_date,
group.my_store AS mys_store,
group.item AS item,
COUNT(list_of.item) AS nb_items;
DUMP result;
当我们将别名项添加到
GROUP BY
语句时,基本上与查找不同的项然后对它们进行计数(就像您在括号中所做的一样)相同。如果您仍然想使用代码,只需在下面的代码中添加一个
food_list.item
关系:result = FOREACH StoreG
{
food_list = FOREACH list_of GENERATE item;
food_count = DISTINCT food_list;
GENERATE FLATTEN(group) AS (category,my_date,my_store),food_list.item,COUNT(food_count);
}
关于hadoop - apache pig 无法执行分组和计数,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34883218/