java - 无法将 Hadoop 2.2 的虚拟内存更改为超过默认的 2.1 GB

Question

我在 CentsOS 6.4 和 8GB RAM 的笔记本电脑上以伪分布式模式运行 hadoop 2.2。

每当我提交作业时，我都会收到一条错误消息，指出虚拟内存使用量已超出，如下所示。

我已将 yarn-site.xml 中的 yarn.nodenamager.vmem-pmem-ratio 比率更改为 10 (10x 1GB) ，但是虚拟内存的增加并没有超过默认的 2.1 GB ，如在下面的错误消息和容器正在被杀死。

有人可以告诉我是否还有其他需要更改的设置吗？提前致谢!

错误信息:

INFO mapreduce.Job: Task Id : attempt_1388632710048_0009_m_000000_2, Status : FAILED
Container [pid=12013,containerID=container_1388632710048_0009_01_000004] is running beyond virtual memory limits. Current usage: 544.9 MB of 1 GB physical memory used; 14.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1388632710048_0009_01_000004 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 12077 12018 12013 12013 (phantomjs) 16 2 1641000960 6728 /usr/local/bin/phantomjs --webdriver=15358 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12013 882 12013 12013 (bash) 1 0 108650496 305 /bin/bash -c /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx200m -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 attempt_1388632710048_0009_m_000000_2 4 1>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004/stdout 2>/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004/stderr 
    |- 12075 12018 12013 12013 (phantomjs) 17 1 1615687680 6539 /usr/local/bin/phantomjs --webdriver=29062 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12074 12018 12013 12013 (phantomjs) 16 2 1641000960 6727 /usr/local/bin/phantomjs --webdriver=5958 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12073 12018 12013 12013 (phantomjs) 17 2 1641000960 6732 /usr/local/bin/phantomjs --webdriver=31836 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12090 12018 12013 12013 (phantomjs) 16 2 1615687680 6538 /usr/local/bin/phantomjs --webdriver=24519 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12072 12018 12013 12013 (phantomjs) 16 1 1641000960 6216 /usr/local/bin/phantomjs --webdriver=10175 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12091 12018 12013 12013 (phantomjs) 17 1 1615687680 6036 /usr/local/bin/phantomjs --webdriver=5043 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12018 12013 12013 12013 (java) 996 41 820924416 79595 /usr/java/jdk1.7.0_25/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/general/hadoop-2.2.0/logs/userlogs/application_1388632710048_0009/container_1388632710048_0009_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 127.0.0.1 56498 attempt_1388632710048_0009_m_000000_2 4
    |- 12078 12018 12013 12013 (phantomjs) 16 3 1615687680 6545 /usr/local/bin/phantomjs --webdriver=12650 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log
    |- 12079 12018 12013 12013 (phantomjs) 17 2 1642020864 7542 /usr/local/bin/phantomjs --webdriver=18444 --webdriver-logfile=/tmp/hadoop-general/nm-local-dir/usercache/general/appcache/application_1388632710048_0009/container_1388632710048_0009_01_000004/phantomjsdriver.log

Container killed on request. Exit code is 143

Answer 1

你测试过改变

yarn.scheduler.maximum-allocation-mb

或

yarn.nodemanager.resource.memory-mb

在 yarn-site.xml 中？

我认为这种属性会有所帮助。检查yarn-default for the keyword "memory"