c++ - AWS CPP TransferManager 与 GetObjectRequest 流到文件 fstream OOM

Question

我正在使用 AWS CPP SDK ( https://github.com/aws/aws-iot-device-sdk-cpp ) 在小型 Linux 系统(只有 32 MB RAM)上从 S3 下载文件。我正在使用 GetObjectRequest 类，如下所示。它运行良好，并将文件下载到我系统上的 FStream，这样它就不会消耗太多 RAM。

现在，我想将下载代码转换为 TransferManager 方法以获得进度回调。我已经重写了那部分代码，它也显示在下面。它开始时很好，打印下载的百分比，但是当它达到 ~14 MB RAM(大约是下载时 Linux 中可用的数量)时，它会因为使用过多的 RAM 而被内核杀死。

我创建了一个文件流，就像我为 GetObjectRequest 所做的那样。我究竟做错了什么？我怎样才能解决这个问题？谢谢。

没有使用所有 RAM 的旧方法:

    // Old way
    GetObjectRequest getObjectRequest;
    getObjectRequest.SetBucket(bucket.c_str());
    getObjectRequest.SetKey(keyName.c_str());
    getObjectRequest.SetResponseStreamFactory([&destination](){
     return Aws::New<Aws::FStream>(
     "s3file", destination, std::ios_base::out); });

    GetObjectOutcome getObjectOutcome = SessionClient->GetObject(getObjectRequest);
    if(getObjectOutcome.IsSuccess())
    {
        std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
    }
    else
    {
        std::cout << "<AWS DOWNLOAD> Get FW failed: " << getObjectOutcome.GetError().GetMessage() << std::endl;
        exit(1);
    }

最终使用过多 RAM 并被内核杀死的新方法:

// New way
Aws::Transfer::TransferManagerConfiguration transferConfig;
transferConfig.s3Client = SessionClient;

std::shared_ptr<Aws::Transfer::TransferHandle> requestPtr(nullptr);

transferConfig.downloadProgressCallback =
        [](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle& handle)
{
    std::cout << "\r" << "<AWS DOWNLOAD> Download Progress: " << static_cast<int>(handle.GetBytesTransferred() * 100.0 / handle.GetBytesTotalSize()) << " Percent " << handle.GetBytesTransferred() << " bytes\n";
};

Aws::Transfer::TransferManager transferManager(transferConfig);

requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&destination](){

    Aws::FStream *stream = Aws::New<Aws::FStream>("s3file", destination, std::ios_base::out);
    stream->rdbuf()->pubsetbuf(NULL, 0);

    return stream; });

requestPtr->WaitUntilFinished();

size_t retries = 0;
//just make sure we don't fail because a download part failed. (e.g. network problems or interuptions)
while (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::FAILED && retries++ < 5)
{
    std::cout << "<AWS DOWNLOAD> FW Download trying download again!" << std::endl;
    transferManager.RetryDownload(requestPtr);
    requestPtr->WaitUntilFinished();
}

// Check status
if ( requestPtr->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED ) {
    if ( requestPtr->GetBytesTotalSize() == requestPtr->GetBytesTransferred() ) {
        std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
        exit(0);
    }
    else {
        std::cout << "<AWS DOWNLOAD> Get FW failed - Bytes downloaded did not equal requested number of bytes: " << requestPtr->GetBytesTotalSize() << requestPtr->GetBytesTransferred() << std::endl;
        exit(1);
    }
}
else {
    std::cout << "<AWS DOWNLOAD> Get FW failed - download was never completed even after retries" << std::endl;
    exit(1);
}

Answer 1

TransferManager 只有在您处于 10mb 或更大的领域并且您想利用并行化时才真正使事情变得更容易。它会预先分配最大堆大小，并且不会增大堆的大小。鉴于您的 RAM 限制，我不会使用 TransferManager。您仍然可以收到进度通知。检查 AmazonWebServiceRequest 类中的回调机制。