我理解广播优化的概念。
当连接中的一侧有小数据时，最好只为小侧进行洗牌。但是为什么不能只使用 executors 进行这种洗牌呢？为什么我们需要使用驱动程序？
如果每个执行者都持有哈希表来映射执行者之间的记录，我认为它应该可以工作。
在 Spark 广播的当前实现中 - 它收集数据给驱动程序，然后对其进行洗牌，对驱动程序的收集操作是我想避免的瓶颈。
关于如何在没有驱动程序内存瓶颈的情况下实现类似优化的任何想法？
最佳答案
你是对的，当前的实现需要在将数据发送到执行器之前将数据收集到驱动程序。
已经有JIRA票SPARK-17556准确解决您的建议:
"Currently in Spark SQL, in order to perform a broadcast join, the driver must collect the result of an RDD and then broadcast it. This introduces some extra latency. It might be possible to broadcast directly from executors."
我从 attached document 复制了建议的解决方案使这个答案自我描述:
"To add a broadcastmethod to RDDto perform broadcastfrom executor, we need some support work as follow:
- Construct BroadCastIdfrom driver, BroadCastManager will supply a method to do this.
// Called from driver to create new broadcast id
def newBroadcastId: Long = nextBroadcastId.getAndIncrement()
BroadCastManagercould be able to create a broadcast with specified id and a persist tag to infer this broadcast isaexecutor broadcast, and its data will be backup onthe hdfs.
In the TorrentBroadcast.writeBlockswrite the block to hdfs, readBlocksread block from local, remote, hdfs by priority.
When construct the Broadcast, we can control whether to upload broadcast data block
BroadCastManagerpost a api to put broadcast data to block manager
关于apache-spark - 为什么广播连接收集数据到驱动程序以洗牌数据？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/67672461/