python-3.x - 错误 : Python3. 4 -> java.lang.RuntimeException : PipeMapRed. waitOutputThreads():子进程失败，代码为 1

Question

我在 Hadoop 上使用 Python mrjob 库时遇到问题。

我搜索了这个错误，但我不知道解决方案。

1. 我做了 chmod +x pythonFile,
2. 插入.py文件的顶部 #!/usr/bin/env python

我的错误看起来像...(这么长)

lim@slave04 ~/python $ python3 MovieRecommender.py -r hadoop --items hdfs:///user/lim/u.data hdfs:///user/lim/u.item > test.txt
No configs found; falling back on auto-configuration
Looking for hadoop binary in /home/lim/hadoop/bin...
Found hadoop binary: /home/lim/hadoop/bin/hadoop
Using Hadoop version 2.6.5
Looking for Hadoop streaming jar in /home/lim/hadoop...
Found Hadoop streaming jar: /home/lim/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar
Creating temp directory /tmp/MovieRecommender.lim.20170604.094652.529281
Copying local files to hdfs:///user/lim/tmp/mrjob/MovieRecommender.lim.20170604.094652.529281/files/...
Running step 1 of 3...
  packageJobJar: [/tmp/hadoop-unjar8911458661886591142/] [] /tmp/streamjob3138735148266254884.jar tmpDir=null
  Connecting to ResourceManager at slave04/127.0.1.1:8035
  Connecting to ResourceManager at slave04/127.0.1.1:8035
  Total input paths to process : 1
  number of splits:2
  Submitting tokens for job: job_1496311320193_0014
  Submitted application application_1496311320193_0014
  The url to track the job: http://slave04:8088/proxy/application_1496311320193_0014/
  Running job: job_1496311320193_0014
  Job job_1496311320193_0014 running in uber mode : false
   map 0% reduce 0%
  Task Id : attempt_1496311320193_0014_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

  Task Id : attempt_1496311320193_0014_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

  Task Id : attempt_1496311320193_0014_m_000001_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

  Task Id : attempt_1496311320193_0014_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1496311320193_0014_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

  Task Id : attempt_1496311320193_0014_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

   map 100% reduce 100%
  Job job_1496311320193_0014 failed with state FAILED due to: Task failed task_1496311320193_0014_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

  Job not successful!
  Streaming Command Failed!
Counters: 17
    Job Counters 
        Data-local map tasks=2
        Failed map tasks=7
        Killed map tasks=1
        Killed reduce tasks=1
        Launched map tasks=8
        Other local map tasks=6
        Total megabyte-milliseconds taken by all map tasks=18734080
        Total megabyte-milliseconds taken by all reduce tasks=0
        Total time spent by all map tasks (ms)=18295
        Total time spent by all maps in occupied slots (ms)=18295
        Total time spent by all reduce tasks (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0
        Total vcore-milliseconds taken by all map tasks=18295
        Total vcore-milliseconds taken by all reduce tasks=0
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
Scanning logs for probable cause of failure...
Looking for history log in hdfs:///tmp/hadoop-yarn/staging...
Looking for history log in /home/lim/hadoop/logs...
Looking for history log in /home/lim/hadoop-2.6.5/logs...
Probable cause of failure:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

Step 1 of 3 failed: Command '['/home/lim/hadoop/bin/hadoop', 'jar', '/home/lim/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar', '-files', 'hdfs:///user/lim/tmp/mrjob/MovieRecommender.lim.20170604.094652.529281/files/MovieRecommender.py#MovieRecommender.py,hdfs:///user/lim/tmp/mrjob/MovieRecommender.lim.20170604.094652.529281/files/mrjob.zip#mrjob.zip,hdfs:///user/lim/tmp/mrjob/MovieRecommender.lim.20170604.094652.529281/files/setup-wrapper.sh#setup-wrapper.sh,hdfs:///user/lim/u.data#u.data', '-input', 'hdfs:///user/lim/u.item', '-output', 'hdfs:///user/lim/tmp/mrjob/MovieRecommender.lim.20170604.094652.529281/step-output/0000', '-mapper', 'sh -ex setup-wrapper.sh python3 MovieRecommender.py --step-num=0 --mapper --items u.data', '-reducer', 'sh -ex setup-wrapper.sh python3 MovieRecommender.py --step-num=0 --reducer --items u.data']' returned non-zero exit status 256

Answer 1

找到安装python3的正确路径:

which python3

假设结果是:

/usr/bin/python3

• 在命令中包含 --python-bin 参数:

  python3 MovieRecommender.py --python-bin /usr/bin/python3 -r hadoop --items hdfs:///user/lim/u.data hdfs:///user/lim/u.item > test.txt
  

或者

• 创建一个包含以下内容的 ~/.mrjob.conf 文件:

  runners:
    hadoop:
      python_bin: /usr/bin/python3
  

  然后用这个命令运行你的程序:

  python3 MovieRecommender.py -r hadoop --items hdfs:///user/lim/u.data hdfs:///user/lim/u.item > test.txt