java - FileService 处理大文件时内存不足

Question

在开发服务器上，我尝试使用 FileService api 将数据写入单个文件。
即使我写入少量数据，关闭文件(使用 close())也不会释放内存。 我需要使用 FileService 中的 closeFinally() 来释放内存。因此，我的猜测是，整个文件都被缓存在内存中，直到调用 closeFinally() 为止，如果我想在单个 blob 中写入一个巨大的文件，这可能会导致一些问题。
有什么办法可以解决这个内存问题吗？

这里是示例代码:

String veryLongString = "verryyyyllooooonnnnggggssssttttrrrriiiiinnnnnggggg";
for (int i = 0; i < 10; ++i) {
    veryLongString += veryLongString;
}
// veryLongString.length() => 51200 char

FileService FILESERVICE =
    FileServiceFactory.getFileService();

// Create new file
String newPath =
    FILESERVICE.createNewBlobFile("application/octet-stream").getFullPath();

for (int i = 0; i < 10; ++i) {
    System.out.println(i);

    // Open file for append
    AppEngineFile file = new AppEngineFile(newPath);
    FileWriteChannel writeChannel = FILESERVICE.openWriteChannel(file, true);
    OutputStream out = Channels.newOutputStream(writeChannel);

    // Append some data to this file
    for (int j = 0; j < 5000; ++j) {
        out.write(veryLongString.getBytes());
    }
    writeChannel.close();
    // Writed 5000*51200 => 256Mo char (to memory, should be
    // also persisted to disk)
}

// Writed all data we want, we finally close the file
AppEngineFile file = new AppEngineFile(newPath);
FileWriteChannel writeChannel = FILESERVICE.openWriteChannel(file, true);
writeChannel.closeFinally();

输出:

0
1
2
3
com.google.apphosting.api.ApiProxy$UnknownException: An error occurred for the API request file.Append().
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:518)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:452)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:430)
    at java.util.concurrent.Executors$PrivilegedCallable$1.run(Executors.java:463)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.concurrent.Executors$PrivilegedCallable.call(Executors.java:460)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
    at java.io.OutputStream.write(OutputStream.java:58)
    at com.google.appengine.api.files.dev.FileMetadata.append(FileMetadata.java:198)
    at com.google.appengine.api.files.dev.LocalFileService.append(LocalFileService.java:357)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:498)
    ... 10 more

Answer 1

开发服务器旨在帮助您测试和调试应用程序；可扩展性并不是它的主要关注点。你是对的，它将文件缓存在内存中；这就是它应该如何工作的。