python - 连接到 dask.distributed 集群时出现 Pickle 错误

Question

这是我的简单代码。尝试运行我的第一个程序。

from dask.distributed import Client
client = Client('192.168.1.102:8786')


def inc(x):
    return x + 1

x = client.submit(inc, 10)
print(x.result())

当尝试使用此命令运行此代码时:

$python3 filename.py

我遇到了这个错误:

/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py
- INFO - Failed to serialize <function inc at 0x7f678ad05840> Traceback (most recent call last):   File
"/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
line 33, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File
 "/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
 line 43, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps' Traceback
 (most recent call last):   File
 "/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
 line 33, in dumps
     return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File "dis.py", line 8, in
 <module>
    x = client.submit(inc, 10)   File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
 643, in submit
     loose_restrictions, priority={skey: 0})   File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line
 1235, in _graph_to_futures
     'tasks': valmap(dumps_task, dsk3),   File "/usr/local/lib/python3.4/dist-packages/toolz/dicttoolz.py", line 84,
 in valmap
     rv.update(zip(iterkeys(d), map(func, itervalues(d))))   File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
 812, in dumps_task
     return {'function': dumps_function(task[0]),   File "/usr/local/lib/python3.4/dist-packages/distributed/worker.py", line
 779, in dumps_function
     b = dumps(func)   File "/usr/local/lib/python3.4/dist-packages/distributed/protocol/pickle.py",
 line 43, in dumps
     return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL) AttributeError: 'module' object has no attribute 'dumps'

Answer 1

通常，pickling 错误表明您的 Dask.distributed 网络的某些元素(工作程序、调度程序或客户端)的 Python 版本不匹配。也许您的工作人员在您没有意识到的情况下在 Python 2 下运行？

这个错误特别指出 cloudpickle 库没有 dumps 方法，这很奇怪。据我所知，cloudpickle 始终具有 dumps 功能。您的环境中是否有一个奇怪的 cloudpickle 库或一个非常旧的版本？

如果您只是想尝试一下，您还可以通过省略调度程序的地址在同一进程中启动本地集群

from dask.distributed import Client

# client = Client('scheduler-address:8786')
client = Client()  # create local "cluster"