hadoop - Pig Latin 中的 Apache Pig 查询所用的 Cpu 时间

Question

Apache Pig Query 执行需要多少时间？ 查询在 Pig Latin 中获取多达 400 万个具有 43 个字段的元组(行)的记录。

A = LOAD '/user/PigTest/year_14/mon_nov/6_sms_03_01.csv' USING PigStorage(',');
bt = foreach A generate $0 as id,$3;
dump bt;
ct = filter bt by id == 3981042 ;
dump ct;
dump MinutesBetween(CurrentTime(),$ti);

并将文件调用为: pig -param ti='date' try.pig

我的系统环境是Linux。

错误是: 错误 1200:不匹配的输入 '(' 期望 RIGHT_PAREN

org.apache.pig.impl.logicalLayer.FrontendException:错误 1000:解析期间出错。不匹配的输入 '(' 期待 RIGHT_PAREN 在 org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1725) 在 org.apache.pig.PigServer$Graph.access$000(PigServer.java:1420) 在 org.apache.pig.PigServer.parseAndBuild(PigServer.java:364) 在 org.apache.pig.PigServer.executeBatch(PigServer.java:389) 在 org.apache.pig.PigServer.executeBatch(PigServer.java:375) 在 org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) 在 org.apache.pig.tools.grunt.GruntParser.parseStopOnError (GruntParser.java:232) 在 org.apache.pig.tools.grunt.GruntParser.parseStopOnError (GruntParser.java:203) 在 org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) 在 org.apache.pig.Main.run(Main.java:608) 在 org.apache.pig.Main.main(Main.java:156) 在 sun.reflect.NativeMethodAccessorImpl.invoke0( native 方法) 在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在 java.lang.reflect.Method.invoke(Method.java:606) 在 org.apache.hadoop.util.RunJar.main(RunJar.java:212) 原因:解析失败:不匹配的输入 '(' 期待 RIGHT_PAREN

Answer 1

   Two problems here
    1. You should print only the relation in DUMP stmt but you are trying to print the function MinutesBetween().
       If you remove the last line the error will be gone.
    2. In command line you are passing 'date' as parameter. In pig 'date' is not a buildin command. so you need to construct the date atleast any one of the format that pig supports.

    Example:
       I am using this date format '2014-11-06T06:01:13' and more date formats are available in the pig docs. you can check it.

    In command line
    >>pig -param ti='2014-11-06T06:01:13' -f try.pig 

    Change the last line of the pig script like this.
    test = FOREACH ct GENERATE MinutesBetween(CurrentTime(),ToDate('$ti'));
    DUMP test;

更新:

创建一个 shell 脚本说 test.sh
1.获取当前时间(即start_time)
2.调用pig脚本(try.pig)
3.获取当前时间(即end_time)
4 获取时间差异并打印它，这样您将获得 pig 脚本实际花费的时间。您可以修改脚本以包括小时和毫秒。

测试.sh

    #!/bin/bash
    START_TIME=$(date +"%s")

    pig -x local try.pig

    END_TIME=$(date +"%s") 
    DIFF=$(($END_TIME-$START_TIME))
    echo "$(($DIFF / 60)) minutes and $(($DIFF % 60)) seconds."

示例输出:

0 minutes and 2 seconds.