amazon-web-services - 运行 EMR 示例，出现 301 错误

Question

我正在尝试运行示例 hadoop-streaming 命令:

hadoop-streaming -files streamingCode/wordSplitter.py \
-mapper wordSplitter.py \
-input s3://elasticmapreduce/samples/wordcount/input \
-output streamingCode/wordCountOut \
-reducer aggregate

但我一直收到这个错误:

Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: Amazon S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 98038E504E150CEC), S3 Extended Request ID: IW1x5otBSepAnPgW/RKELCUI9dhADQvrXqU2Ase1CLIa0SWDFnBbTscXihrvHvNm2ZRxjjSJZ1Q=

我认为这是因为我的集群在 us-west-2 中，但我不知道如何正确格式化 s3 url(或者也许这根本不是问题)。

编辑:改成如下url后:

s3://s3-us-west-2.amazonaws.com/elasticmapreduce/samples/wordcount/input

我现在收到以下错误:

Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3
Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: BC8DB415C780DF84), 
           S3 Extended Request ID: sx8W/+gvND2ssqQce9ZQsZTiqxmSJYZs8OiXgrjwL3dm0JRPaC7ceHor+yrHsPuKTjM2LUwkRAw=

编辑:所以我已经确认错误确实是因为我的集群在us-west-2，我在us-east-1中创建了一个集群它工作正常。那么，问题是如何从另一个区域访问 s3 存储桶？这可能吗？

Answer 1

Amazon 更改了从 emr-4.7.0 开始的默认行为，这在我们升级 EMR 版本时导致了此错误。

解决方法很简单，在core-site中添加如下配置: fs.s3n.endpoint=s3.amazonaws.com