关于 FileOutputStream 的 java OutOfMemoryError？

Question

谢谢大家^_^，问题解决了:有一行太大了(超过400M...不知不觉下载了损坏的文件)，所以抛出OutOfMemoryError

我想用java分割一个文件，但是总是抛出OutOfMemoryError: Java heap space，找遍了全网，好像没有什么帮助:(

附言。文件大小600M，超过3000万行，每行不超过100个字符。 (也许你可以像这样生成一个“关卡文件”:{ 编号:0000000001，等级:1 编号:0000000002，等级:2 ....(超过 3000 万) })

嘘。将 Jvm 内存大小设置得更大是行不通的，:(

嘘。换了一台电脑，问题依旧/(ㄒoㄒ)/~~

无论我设置的 -Xms 或 -Xmx 有多大，输出文件的大小总是相同的，(并且 Runtime.getRuntime().totalMemory() 确实改变了)

这是堆栈跟踪:

 Heap Size = 2058027008
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
        at java.lang.StringBuffer.append(StringBuffer.java:306)
        at java.io.BufferedReader.readLine(BufferedReader.java:345)
        at java.io.BufferedReader.readLine(BufferedReader.java:362)
        at com.xiaomi.vip.tools.ptupdate.updator.Spilt.main(Spilt.java:39)
    ...

这是我的代码:

package com.updator;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;

public class Spilt {
    public static void main(String[] args) throws Exception {
        long heapSize = Runtime.getRuntime().totalMemory();

        // Print the jvm heap size.
        System.out.println("Heap Size = " + heapSize);

        String mainPath = "/home/work/bingo/";
        File mainFilePath = new File(mainPath);
        FileInputStream inputStream = null;
        FileOutputStream outputStream = null;
        try {
            if (!mainFilePath.exists())
                mainFilePath.mkdir();

            String sourcePath = "/home/work/bingo/level.txt";
            inputStream = new FileInputStream(sourcePath);
            BufferedReader bufferedReader = new BufferedReader(new FileReader(
                    new File(sourcePath)));

            String savePath = mainPath + "tmp/";
            Integer i = 0;
            File file = new File(savePath + "part"
                    + String.format("%0" + 5 + "d", i) + ".txt");
            if (!file.getParentFile().exists())
                file.getParentFile().mkdir();
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            int count = 0, total = 0;
            String line = null;
            while ((line = bufferedReader.readLine()) != null) {
                line += '\n';
                outputStream.write(line.getBytes("UTF-8"));
                count++;
                total++;
                if (count > 4000000) {
                    outputStream.flush();
                    outputStream.close();
                    System.gc();
                    count = 0;
                    i++;
                    file = new File(savePath + "part"
                            + String.format("%0" + 5 + "d", i) + ".txt");
                    file.createNewFile();
                    outputStream = new FileOutputStream(file);
                }
            }

            outputStream.close();
            file = new File(mainFilePath + "_SUCCESS");
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            outputStream.write(i.toString().getBytes("UTF-8"));
        } finally {
            if (inputStream != null)
                inputStream.close();
            if (outputStream != null)
                outputStream.close();
        }
    }
}

---

我想可能是:outputStream.close()的时候，内存没有释放？

Answer 1

因此您打开原始文件并创建一个 BufferedReader 和一个行计数器。

char[] buffer = new char[5120];
BufferedReader reader = Files.newBufferedReader(Paths.get(sourcePath), StandardCharsets.UTF_8);
int lineCount = 0;

现在您读入您的缓冲区，并在字符进入时将其写入。

int read;

BufferedWriter writer = Files.newBufferedWriter(Paths.get(fileName), StandardCharsets.UTF_8);
while((read = reader.read(buffer, 0, 5120))>0){
    int offset = 0;
    for(int i = 0; i<read; i++){
        char c = buffer[i];
        if(c=='\n'){
           lineCount++;
           if(lineCount==maxLineCount){
              //write the range from 0 to i to your old writer.
              writer.write(buffer, offset, i-offset);
              writer.close();
              offset=i;
              lineCount=0;
              writer = Files.newBufferedWriter(Paths.get(newName), StandarCharset.UTF_8);
           }
        }
        writer.write(buffer, offset, read-offset);
    }
    writer.close();
}

这应该会降低内存使用率并防止您一次读取太多行。您可以不使用 BufferedWriters 并进一步控制内存，但我认为这没有必要。