python - Mrjob 无法在 dataproc 上创建集群 : __init__() got an unexpected keyword argument 'channel'

Question

我正在尝试使用 Python mrjob 库在 Google Cloud Dataproc 上运行 Hadoop Map Reduce 字数统计示例。但是，mrjob 失败并出现以下异常:

TypeError: __init__() got an unexpected keyword argument 'channel'
Traceback (most recent call last):
  File "freq.py", line 21, in <module>
    MRWordFreqCount.run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 616, in run
    cls().execute()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 687, in execute
    self.run_job()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 636, in run_job
    runner.run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/runner.py", line 503, in run
    self._run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 468, in _run
    self._launch()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 473, in _launch
    self._launch_cluster()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 637, in _launch_cluster
    self._get_cluster(self._cluster_id)
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 1188, in _get_cluster
    return self.cluster_client.get_cluster(
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 376, in cluster_client
    return google.cloud.dataproc_v1beta2.ClusterControllerClient(
TypeError: __init__() got an unexpected keyword argument 'channel'

我检查了 GOOGLE_APPLICATION_CREDENTIALS 是否设置正确，API 是否已在 Google Cloud 上启用，并且是否已为服务帐号设置所有必需的角色。

mrjob 成功将文件上传到 Google Cloud Storage。但一旦尝试创建新的 Dataproc 集群就会失败。

可能出现什么问题？

在 Dataproc 上启动 mrjob 的命令行:

$ python3 freq.py -r dataproc words.txt

当前Python环境:

$ python3 -VV
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]

$ pip3 list | grep google
google-api-core          1.23.0
google-auth              1.23.0
google-auth-oauthlib     0.4.2
google-cloud-core        1.4.3
google-cloud-dataproc    2.0.2
google-cloud-logging     1.15.1
google-cloud-storage     1.32.0
google-crc32c            1.0.0
google-pasta             0.2.0
google-resumable-media   1.1.0
googleapis-common-protos 1.52.0

$ pip3 list | grep mrjob
mrjob                    0.7.4

Answer 1

解决方案是将 google-cloud-dataproc 降级到 1.1.1。

调试 mrjob 实现后，我发现 mrjob 版本 0.7.4 使用自 2.0.0 版以来在 google-cloud-dataproc 库上重命名的参数调用 google.cloud.dataproc_v1beta2.ClusterControllerClient 的构造函数。

如何使用 pip3 降级:

$ pip3 install --force-reinstall --no-deps google-cloud-dataproc==1.1.1