python - 在本地集群上运行 dask 时超时 OSError

Question

我正在尝试在带有配置的 Power PC 上运行以下代码:

Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.6:GA:server
          Kernel: Linux 3.10.0-957.21.3.el7.ppc64le
    Architecture: ppc64-le

具有 20 个核心的单节点本地集群。

import os, subprocess
from timeit import default_timer as timer
from dask.distributed import Client, LocalCluster, fire_and_forget, as_completed

def run_client(n_workers):
    files = []
    for dirpaths, dirnames, filenames in os.walk('cap_logs/'):
        if not dirnames:
            files.extend([os.path.join(dirpaths, file) for file in filenames])

    def parser(file):
        val = subprocess.run(['./test.sh', file], stdout=subprocess.PIPE)
        return val.stdout.decode()

    cluster = LocalCluster(n_workers=n_workers, dashboard_address=None)
    with Client(cluster) as client:
        futures = []
        files = client.scatter(files)
        futures = client.map(parser, files)
        results = [future.result() for future in as_completed(futures)]
        del futures
        cluster.close()

workers = [20, 18, 16, 14, 12, 10, 8, 7, 6, 5, 4, 3, 2, 1]
times = {}
for n_workers in workers:
    tic = timer()
    run_client(n_workers)
    toc = timer()
    time = toc - tic
    times[n_workers] = round(time, 2)

如果 n_workers 相对小于 (<15) 核心总数即 20，它工作正常，但一旦我将 n_workers 设置为 >15，它就会出现以下错误:

OSError: Timed out trying to connect to 'tcp://127.0.0.1:34487' after 10 s: connect() didn't finish in time

Answer 1

我很惊讶你看到如此少的 worker 超时。但即便如此，您可能想尝试为 dask 配置的 distributed.timeouts 部分提供更长的 connect 超时:

distributed:
  comm:
    timeouts:
      connect: 10s          # time before connecting fails
      tcp: 30s              # time before calling an unresponsive connection dead

full default config可以在源代码中找到。