apache-spark - DataProc集群Spark作业提交无法启动NodeManager

Question

我们的 Dataproc 集群配置了 4 个工作线程。集群已启动并正在运行，每当我们尝试提交 Spark 作业时，我们都会收到此错误:

YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager, Registration of NodeManager failed, Message from ResourceManager: Disallowed NodeManager

Stackdriver 日志中看到的一些消息是

Daemon YARN_NODE_MANAGER failed to restart

更新: 即使我们向现有 Dataproc 集群添加新的工作节点，也会注意到此问题。

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager, Registration of NodeManager failed, Message from ResourceManager: Disallowed NodeManager from <MasterNode DNS> , Sending SHUTDOWN signal to the NodeManager.
    at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:374)
    at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:252)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:845)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:912)

Answer 1

此错误看起来像是 YARN 节点管理器停用问题。您能否检查一下Dataproc master GCE VM中以下YARN包含/排除节点配置文件是否有错误:

• /etc/hadoop/conf/nodes_exclude
• /etc/hadoop/conf/nodes_include

更改这些配置文件后，请运行刷新节点命令:

yarn rmadmin -refreshNodes 

然后您应该会看到 Nodemanager 重新加入 YARN。

详情请参阅: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html#nodeslistmanager-detects-and-handles-include-and-exclude-list-changes