java - 在 S3 中迭代对象时为 "ConnectionPoolTimeoutException"

Question

我已经使用 aws java API 工作了一段时间，没有遇到太多问题。目前我正在使用库 1.5.2 版本。

当我使用以下代码迭代文件夹中的对象时:

AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(MyClass.class.getResourceAsStream("AwsCredentials.properties")));

String s3Key = "folder1/folder2";


String bucketName = Constantes.S3_BUCKET;
String key = s3Key +"/input_chopped/";

ObjectListing  current = s3.listObjects(new ListObjectsRequest()
        .withBucketName(bucketName)
        .withPrefix(key));

boolean siguiente  = true;

while (siguiente) {    

    siguiente &= current.isTruncated();
    contador += current.getObjectSummaries().size();

    for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));
        System.out.println(object.getKey());
    }

    current=s3.listNextBatchOfObjects(current);

}

要点:链接:https://gist.github.com/fgblanch/6038699 我收到以下异常:

INFO  (AmazonHttpClient.java:358) - Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
    at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:315)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2994)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:918)
    at com.madiva.segmentacion.tests.ListaS3.main(ListaS3.java:177)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
Error Message: Unable to execute HTTP request: Timeout waiting for connection from pool

知道如何避免这个错误。它只发生在有多个对象的文件夹中，在这种情况下，里面有 463 个文件。谢谢

Answer 1

我发现 S3Object 为每个对象打开一个连接。即使对象被垃圾收集也不会被释放，因此需要执行 object.close()，以释放与池的连接。

所以更正后的代码是:

 for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));             
        System.out.println(object.getKey());
        object.close();
    }