python - gae mapreduce 生成器错误 无属性 validate_bucket_name

Question

这是我的第一个 GAE 项目。我的串行代码可以在 dev_app 上运行(我在 Mac 上使用 GoogleAppEngineLauncher)。由于我的代码需要很长时间才能完成，我尝试使用 MapReduce 来加速该过程。我尝试了以下代码，但不断收到以下错误。我不确定这是否是因为我的代码中存在一些错误，或者我是否缺少 *yaml 文件中的任何语句。请帮忙!

class ShuffleDictPipeline(base_handler.PipelineBase):
  def run(self, *args, **kwargs):
    """ run """
    mapper_params = {
        "entity_kind": "coremic.RandomDict",
        "batch_size": 500,
        "filters": [("idx", "=", ndb_custom_key)]
    }
    reducer_params = {
        "mime_type": "text/plain"
    }
    output = yield mapreduce_pipeline.MapreducePipeline(
        "calc_shuff_core_microb",
        mapper_spec="coremic.shuffle_dict_coremic_map",
        mapper_params=mapper_params,
        reducer_spec="coremic.shuffle_dict_coremic_reduce", 
        reducer_params=reducer_params,
        input_reader_spec="mapreduce.input_readers.DatastoreInputReader", 
        output_writer_spec="mapreduce.output_writers.BlobstoreOutputWriter", 
       shards=16)

    yield StoreOutput(output)

错误:

ERROR    2016-03-05 20:03:21,706 pipeline.py:2432] 
Generator mapreduce.mapper_pipeline.MapperPipeline(*(u'calc_shuff_core_microb-map', u'coremic.shuffle_dict_coremic_map', u'mapreduce.input_readers.DatastoreInputReader'), **{'output_writer_spec': u'mapreduce.output_writers._GoogleCloudStorageKeyValueOutputWriter', 'params': {u'batch_size': 500, u'bucket_name': u'app_default_bucket', u'entity_kind': u'coremic.RandomDict',... (324 bytes))#b96dd511c0454fd99413d267b7388857 raised exception. AttributeError: 'NoneType' object has no attribute 'validate_bucket_name'

Traceback (most recent call last):
  File "/Users/rr/GAE/coremic/pipeline/pipeline.py", line 2156, in evaluate
self, pipeline_key, root_pipeline_key, caller_output)
  File "/Users/rr/GAE/coremic/pipeline/pipeline.py", line 1110, in _run_internal
    return self.run(*self.args, **self.kwargs)
  File "/Users/rr/GAE/coremic/mapreduce/mapper_pipeline.py", line 102, in run
queue_name=self.queue_name,
  File "/Users/rr/GAE/coremic/mapreduce/control.py", line 125, in start_map
in_xg_transaction=in_xg_transaction)
  File "/Users/rr/GAE/coremic/mapreduce/handlers.py", line 1730, in _start_map
mapper_output_writer_class.validate(mapper_spec)
  File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 1075, in validate
return cls.WRITER_CLS.validate(mapper_spec)
  File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 723, in validate
super(_GoogleCloudStorageOutputWriter, cls).validate(mapper_spec)
  File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 604, in validate
cloudstorage.validate_bucket_name(
AttributeError: 'NoneType' object has no attribute 'validate_bucket_name'

Answer 1

我仍在努力让一切正常运转，但有几件事有所帮助。

1.1 在 SDK 上安装 google 云存储客户端 lib 以访问存储桶。云 google com appengine 文档 python googlecloudstorageclient

1.2 设置(创建)存储桶。

然后按照 https://plus.google.com/+EmlynORegan/posts/6NPaRKxMkf3 中的步骤操作
请注意映射器参数如何更改。

2 - 在mapreduce管道中，替换 “mapreduce.output_writers.BlobstoreOutputWriter” 和 “mapreduce.output_writers.GoogleCloudStorageConsistentOutputWriter”

3 - 将 reducer 参数更新为:

{ "mime_type": "文本/纯文本", “输出作家”:{ “桶名称”:， “tmp_bucket_name”: } }

其他非常有用的链接:
https://gist.github.com/nlathia/ab670053ed460c4ca02f/89178e132b894fe5467c09164d3827f70e4ae2f8