我正在尝试在 Hadoop 上执行 map reduce 程序。
当我将作业提交到 hadoop 单节点集群时。正在创建作业,但失败并显示消息
“容器被 ApplicationMaster 杀死”
使用的输入大小为 10 MB。
当我使用输入文件 400 KB 的相同脚本时,它成功了。但是对于大小为 10 MB 的输入文件失败。
我的终端显示的完整日志如下。
15/05/29 09:52:16 WARN util.NativeCodeLoader: Unable to `load native- hadoop library for your platform... using builtin-java classes where applicable
Submitting job on the cluster...
15/05/29 09:52:17 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/05/29 09:52:18 INFO input.FileInputFormat: Total input paths to process : 1
15/05/29 09:52:18 INFO mapreduce.JobSubmitter: number of splits:1
15/05/29 09:52:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1432910768528_0001
15/05/29 09:52:19 INFO impl.YarnClientImpl: Submitted application application_1432910768528_0001
15/05/29 09:52:19 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1432910768528_0001/
15/05/29 09:52:19 INFO mapreduce.Job: Running job: job_1432910768528_0001
15/05/29 09:52:29 INFO mapreduce.Job: Job job_1432910768528_0001 running in uber mode : false
15/05/29 09:52:29 INFO mapreduce.Job: map 0% reduce 0%
15/05/29 09:52:41 INFO mapreduce.Job: map 100% reduce 0%
15/05/29 10:03:01 INFO mapreduce.Job: map 0% reduce 0%
15/05/29 10:03:01 INFO mapreduce.Job: Task Id : attempt_1432910768528_0001_m_000000_0, Status : FAILED
AttemptID:attempt_1432910768528_0001_m_000000_0 Timed out after 600 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
我这里的映射器正在触发另一个程序,它将在这里处理我的输入文件。被映射器触发的程序通常会消耗大量内存。
所以请在这方面帮助我。
最佳答案
在 yarn-site.xml
中包含以下属性并重启 VM
,
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>
关于java - Hadoop 映射器因 "Container killed by the ApplicationMaster"而失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/30533501/