我的 spark Job 是由 oozie in hue 提交的。 Spark 在 yarn 集群模式下运行。我正在尝试通过所谓的驱动程序的4040端口来监控正在运行的应用程序状态,但是我找不到4040端口,我检查了进程:
appuser 137872 137870 0 18:55 ? 00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx4096m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp '-Dspark.driver.port=36503' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.117.107:36503 --executor-id 3 --hostname 10.120.117.100 --cores 1 --app-id application_1493800575189_0547 --user-class-path file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar 1> /home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stdout 2> /home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stderr
appuser 138337 137872 99 18:55 ? 00:05:11 /home/jdk/bin/java -server -Xmx4096m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp -Dspark.driver.port=36503 -Dspark.ui.port=0 -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004 -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.117.107:36503 --executor-id 3 --hostname 10.120.117.100 --cores 1 --app-id application_1493800575189_0547 --user-class-path file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar
我不知道为什么 spark.ui.port 是 0 而不是 4040。当然,我的 linux 系统不允许使用端口 0。所以,我无法从 REST api 监控应用程序状态。
谁能给我一些建议?
非常感谢 Mariusz 的回答,下面是 spark ApplicationMaster 进程吗?
[appuser@hz-10-120-117-100 bin]$ ps -ef|grep ApplicationMaster
appuser 125805 125803 0 May03 ? 00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0014/container_1493800575189_0014_01_000001/tmp -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'com.netease.ecom.data.gjs.statis.online.app.day.AppDayRealtimeStatis' --jar hdfs://datahdfsmaster/user/appuser/bjmazhengbing/jar/spark_streaming/spark-streaming-etl-2.0.jar --arg 'analysis_gjs_online.properties' --arg 'rrr' --properties-file /data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0014/container_1493800575189_0014_01_000001/__spark_conf__/__spark_conf__.properties 1> /home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001/stdout 2> /home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001/stderr
根据Spark的官方文档,驱动程序应该有一个4040端口用于监控,但是我的驱动程序似乎没有打开任何端口:
[appuser@hz-10-120-117-100 bin]$ netstat -ntlp|grep 125805
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
我最终找出驱动端口的目的是监控应用程序状态。有什么建议吗?
最佳答案
您列出的进程是执行程序,而不是驱动程序。
当你以 yarn-cluster
模式运行应用程序时,spark driver 和 yarn application master 运行在同一个 JVM 中。因此,确定 Spark UI 地址的最简单方法是转到资源管理器的 UI,找到您的应用程序并单击指向 Application Master 的链接。这将是一个代理地址,指向驱动程序的 ui 端口。
关于apache-spark - 如何找到我的 Spark 的驱动程序节点?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/43759186/