amazon-web-services - 无法使用 Hadoop 访问 S3 存储桶

Question

我正在尝试使用 Hadoop (2.7.3) 访问我的 S3 存储桶，我得到以下信息

ubuntu@AWS:~/Prototype/hadoop$ ubuntu@AWS:~/Prototype/hadoop$ bin/hadoop fs -ls s3://[bucket]/

17/03/24 15:33:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable -ls: Fatal internal error com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 1FA2318A386330C0, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: 1S7Eq6s9YxUb9bPwyHP73clJvD619LZ2o0jE8VklMAA9jrKXPbvT7CG6nh0zeuluGrzybiPbgRQ= at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) ubuntu@AWS:~/Prototype/hadoop$

conf-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>s3://[ Bucket ]</value>
    </property>

    <property>
            <name>fs.s3a.endpoint</name>
            <value>s3.eu-central-1.amazonaws.com</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3a.secret.key</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3.awsAccessKeyId</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3.awsSecretAccessKey</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3n.awsAccessKeyId</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3n.awsSecretAccessKey</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>

    <!-- Comma separated list of local directories used to buffer
         large results prior to transmitting them to S3. -->
    <property>
        <name>fs.s3.buffer.dir</name>
        <value>/tmp</value>
    </property>
</configuration>

有人知道问题出在哪里吗？

编辑:存储桶和访问它的虚拟机位于法兰克福。它似乎类似于 https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-trouble/index.html但在添加端点后它仍然不起作用。

Answer 1

听起来像是 V4 身份验证问题，fs.s3a.endpoint 属性应该已经解决了这个问题

时钟问题也可能导致问题。检查 Joda 时间，并确保您所有的机器都 catch 了本周末的时钟更改。

也尝试获取 Hadoop 2.8.0 RC3，看看问题是否已经消失。如果它仍然存在，那就是在 apache 列表中寻求帮助的版本。