python - 如何在 Amazon S3 中复制大于 5 GB 的文件？

Question

Amazon S3 REST API 文档说在 PUT 操作中上传的大小限制为 5gb。大于该大小的文件必须使用 multipart 上传。很好。

但是，我本质上需要的是重命名可能比这更大的文件。据我所知，没有重命名或移动操作，因此我必须将文件复制到新位置并删除旧位置。大于 5gb 的文件究竟是如何完成的？我必须从存储桶到自身进行分段上传吗？在这种情况下，如何将文件分成几部分？

从阅读 boto 的源代码来看，对于大于 5gb 的文件，它似乎不会自动执行类似的操作。有没有我错过的内置支持？

Answer 1

As far as I know there's no rename or move operation, therefore I have to copy the file to the new location and delete the old one.

没错，对于小于 5 GB 的对象/文件，通过 PUT Object - Copy 很容易做到这一点。操作，后跟 DELETE Object操作(当然，boto 都支持这两者，见 copy_key() 和 delete_key() ):

This implementation of the PUT operation creates a copy of an object that is already stored in Amazon S3. A PUT copy operation is the same as performing a GET and then a PUT. Adding the request header, x-amz-copy-source, makes the PUT operation copy the source object into the destination bucket.

但是，对于大于 5 GB 的对象/文件，这确实是不可能的:

Note
[...] You create a copy of your object up to 5 GB in size in a single atomic operation using this API. However, for copying an object greater than 5 GB, you must use the multipart upload API. For conceptual information [...], go to Uploading Objects Using Multipart Upload [...] [emphasis mine]

同时，Boto 也通过 copy_part_from_key() 支持这一点。方法;不幸的是，所需的方法没有记录在相应的 pull request #425 (allow for multi-part copy commands) 之外。 (虽然我自己还没有尝试过):

import boto
s3 = boto.connect_s3('access', 'secret')
b = s3.get_bucket('destination_bucket')
mp = b.initiate_multipart_upload('tmp/large-copy-test.mp4')
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 1, 0, 999999999)
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 2, 1000000000, 1999999999)
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 3, 2000000000, 2999999999)
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 4, 3000000000, 3999999999)
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 5, 4000000000, 4999999999)
mp.copy_part_from_key('source_bucket', 'path/to/source/key', 6, 5000000000, 5500345712)
mp.complete_upload()

您可能想研究有关如何最终在 Java 或 .NET 中实现此目的的相应示例，这可能会提供对一般方法的更多见解，请参阅 Copying Objects Using the Multipart Upload API .

祝你好运!

---

附录

请注意以下一般复制的特殊性，很容易被忽视:

When copying an object, you can preserve most of the metadata (default) or specify new metadata. However, the ACL is not preserved and is set to private for the user making the request. To override the default ACL setting, use the x-amz-acl header to specify a new ACL when generating a copy request. For more information, see Amazon S3 ACLs. [emphasis mine]