apache-spark - NettyBlockTransferService不遵守spark.blockManager.port配置

Question

我正在运行Spark 2.4.4。在纱上。 NodeManager上的spark配置如下所示:

spark-defaults.conf:

spark.driver.port=38429
spark.blockManager.port=35430
spark.driver.blockManager.port=44349

创建Spark驱动程序和执行程序时，它们会选择驱动程序端口(38429)配置，而不是blockManager(35430)/ driver.blockManager(44349)配置。 blockManager端口是随机分配的

司机:

14:23:40 INFO spark.SparkContext: Running Spark version 2.4.4
14:23:40 INFO util.Utils: Successfully started service 'sparkDriver' on port **38429**.
14:23:41 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38171.
14:23:41 INFO netty.NettyBlockTransferService: Server created on driverhost:**38171**

执行者:

14:23:44 INFO client.TransportClientFactory: Successfully created connection to driverhost:**38429** after 73 ms (0 ms spent in bootstraps)
14:23:45 INFO executor.Executor: Starting executor ID 1 on host ...
14:23:45 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34914.
14:23:45 INFO netty.NettyBlockTransferService: Server created on executorhost:**34914**

我遇到了一个描述该问题的错误Jira，但它是针对Spark 2.4.0提出的，并在12个月前关闭:https://issues.apache.org/jira/browse/SPARK-27139

查看GitHub中的Spark代码，我看不到任何明显的东西:

https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/SparkEnv.scala

333    val blockManagerPort = if (isDriver) {
334      conf.get(DRIVER_BLOCK_MANAGER_PORT)
335    } else {
336      conf.get(BLOCK_MANAGER_PORT)
337    }
338
339    val blockTransferService =
340      new NettyBlockTransferService(conf, securityManager, bindAddress, advertiseAddress,
341        blockManagerPort, numUsableCores)

https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/internal/config/package.scala

308  private[spark] val BLOCK_MANAGER_PORT = ConfigBuilder("spark.blockManager.port")
309    .doc("Port to use for the block manager when a more specific setting is not provided.")
310    .intConf
311    .createWithDefault(0)
312
313  private[spark] val DRIVER_BLOCK_MANAGER_PORT = ConfigBuilder("spark.driver.blockManager.port")
314    .doc("Port to use for the block manager on the driver.")
315    .fallbackConf(BLOCK_MANAGER_PORT)

谁能告诉我为什么我的NettyBlockTransferService端口是随机分配的，而不是35430或44349？

Answer 1

这里的问题是在YARN NodeManager上设置此配置。需要在客户端上进行设置，即在提交Spark应用程序的过程中进行设置，而不是在集群本身上进行设置。