我有以下方法可以分段上传到 amazonS3:
public static void uploadToS3UserPath(String filePath, String bucket, String userPath) {
ClientConfiguration config = new ClientConfiguration();
config.setProxyHost("host");
config.setProxyPort(3128);
config.setProtocol(Protocol.HTTP);
TransferManager tm = new TransferManager(new AmazonS3Client(credentials, config));
TransferManagerConfiguration conf = new TransferManagerConfiguration();
conf.setMinimumUploadPartSize(50 * 1024 * 1024); //use 50 megabytes parts;
tm.setConfiguration(conf);
PutObjectRequest req = new PutObjectRequest(bucket, userPath, new File(filePath));
req.setCannedAcl(CannedAccessControlList.PublicRead);
req.setStorageClass(StorageClass.ReducedRedundancy); //we do it to decrease storage costs
Upload up = tm.upload(req);
try {
up.waitForCompletion();
tm.shutdownNow();
System.out.println("Upload completed successfully");
} catch (AmazonClientException ex) {
Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
tm.shutdownNow();
} catch (InterruptedException ex) {
Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
}
}
有时候我会得到:
com.amazonaws.AmazonClientException: Unable to complete transfer: null
at
com.amazonaws.services.s3.transfer.Transfer.unwrapExecutionException(Transfer.java:226)
at
com.amazonaws.services.s3.transfer.Transfer.rethrowExecutionException(Transfer.java:210)
at
com.amazonaws.services.s3.transfer.Transfer.waitForCompletion(Transfer.java:116)
at
xx.xxx.xxx.xxx.agent.Utilities.uploadToS3UserPath(Utilities.java:373)
at
xx.xxx.xxx.xxx.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:312)
at
uk.org.infectogenomics.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:305)
at xx.xxx.xxx.xxx.hostEl.HostEl.run(HostEl.java:598)
at xx.xxx.xxx.xxx.agent.Agent.main(Agent.java:98)
Caused by: java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:216)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:378)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.reschedule(UploadMonitor.java:210)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:197)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:148)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:49)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
问题是我不知道为什么这只是为了随机上传。明明是RejectedExecutionException的原因?会不会是因为我过早地关闭了 TransferManager?
最佳答案
Could it be because I'm shuttingdown the TransferManager prematurely?
这确实是此行为的诱因,但根本原因是 AWS SDK for Java 中的错误, 参见 Eric Milas 类似问题 Amazon AWS Java SDK TransferManager issue :
TransferManager contains an UploadMonitor and UploadMonitor has a static ScheduledExecutorService called timedThreadPoool. TransferManager.shutdownNow() calls the static method UploadMonitor.shutdownNow() which calls timedThreadPoool.shutdownNow(). The next time any call to UploadMonitor.reschedule() is made the RejectedExecutionException is thrown.
His own answer引用 AWS 团队对 TransferManager Thread Issue 的响应,它确认了此错误,并且自 Release 1.3.2 起已实现相应的修复程序):
The Amazon S3 TransferManager class was updated to address two bugs:
Possible data corruption when uploading from an InputStream and encountering IO errors during certain portions of the upload. Cleanly shutting down timer threads when shutdownNow() is called on an instance of TransferManager
关于java - 使用 AWS TransferManager 时出现 RejectedExecutionException,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/7688004/