我是 Hadoop 的绝对初学者，我只是在做一些简单的测试，但是，我发现错误消息并没有提供非常有用的信息。
我在 CentOS 6.4 VM 上以单节点模式设置了我的 Hadoop 环境，有 4Gb 可用内存。
我正在尝试在一个 500Mb CSV 文件上运行一个简单的 Pig 脚本。我有两个 500Mb 的文件，在第一个上，脚本成功了。在第二个上，它的大小大致相同，但数据不同(行数更多)，当执行达到大约 60% 时出现错误。
这是我使用的(非常简单的)Pig 脚本:
records = LOAD 'trans2013.csv' USING PigStorage(',') AS
(podracun_v_breme,datum_transakcije,znesek_transakcije,oznaka_valute_transakcije,racun_v_dobro,naziv_prejemnika,maticna_stevilka,davcna_stevilka,sifra_pu,zr_sns_oe,namen);
transaction_recs = GROUP records ALL;
tot_trans = FOREACH transaction_recs GENERATE
SUM(records.znesek_transakcije);
STORE tot_trans INTO '/user/root/totaltransactions';
这是我在终端中得到的错误:
2014-04-06 10:28:29,147 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 64% complete 2014-04-06 10:28:30,240 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1396637732046_0008 has failed! Stop running all dependent jobs 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2014-04-06 10:28:30,460 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
2014-04-06 10:28:30,461 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! 2014-04-06 10:28:30,463 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.0.6-alpha 0.11.1 root 2014-04-06 10:25:49 2014-04-06 10:28:30 GROUP_BY
Failed!
Failed Jobs: JobId Alias Feature Message Outputs job_1396637732046_0008 records,tot_trans,transaction_recs GROUP_BY,COMBINER Message: Job failed! /user/root/totaltransactions,
Input(s): Failed to read data from "hdfs://localhost:8020/user/root/trans2013.csv"
Output(s): Failed to produce result in "/user/root/totaltransactions"
Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0
Job DAG: job_1396637732046_0008
2014-04-06 10:28:30,463 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2014-04-06 10:28:30,491 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
Details at logfile: /root/pig_1396797945352.log
这是日志中的错误:
Backend error message --------------------- AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
Pig Stack Trace --------------- ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:149) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:400) at org.apache.pig.PigServer.launchPlan(PigServer.java:1264) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249) at org.apache.pig.PigServer.execute(PigServer.java:1239) at org.apache.pig.PigServer.executeBatch(PigServer.java:333) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:604) at org.apache.pig.Main.main(Main.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
最佳答案
所以...我尝试使用 mapreduce 选项运行 Pig 脚本:
pig -x mapreduce script.pig
它仍然失败了，但它至少产生了一个有意义的错误。看来我不得不从 CSV 文件中删除标题，因为 Pig 使用标题行作为数据。这似乎只在处理 float 时才会发生——如果对整数使用相同的脚本，标题行将被忽略。
原来如此。首先从文件中删除 header ，然后针对它运行脚本 - 它起作用了。
关于Hadoop: pig 错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/22896220/