memory - 声明非常大的数组并迭代 stdin 时内存持续增加

Question

以下代码声明了两个数组，然后遍历 stdin(只是盲目地遍历文件 - 不与数组交互)。

这导致内存不断增加。

但是，如果我只声明两个数组并休眠 - 内存不会增加。

同样，如果我只是迭代 stdin - 内存不会增加。

但加起来(除了分配给数组的内存)还有一个持续的增长。

我通过使用 top 工具查看 RES 内存来测量这一点。

我把func doSomething()中的前几行注释掉了，说明注释的时候没有内存增加。取消注释行并运行将导致增加。

NOTE: This was run on go 1.4.2, 1.5.3 and 1.6

NOTE: You will need to recreate this on a machine with at least 16GB RAM as I have observed it only on the array size of 1 billion.

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

type MyStruct struct {
    arr1 []int
    arr2 []int
}

func (ms *MyStruct) Init(size int, arr1 []int, arr2 []int) error {
    fmt.Printf("initializing mystruct arr1...\n")
    ms.arr1 = arr1
    if ms.arr1 == nil {
        ms.arr1 = make([]int, size, size)
    }
    fmt.Printf("initializing mystruct arr2...\n")
    ms.arr2 = arr2
    if ms.arr2 == nil {
        ms.arr2 = make([]int, size, size)
    }
    fmt.Printf("done initializing ...\n")
    for i := 0; i < size; i++ {
        ms.arr1[i] = 0
        ms.arr2[i] = 0
    }
    return nil
}

func doSomething() error {
    fmt.Printf("starting...\n")
    fmt.Printf("allocating\n")
    /* NOTE WHEN UNCOMMENTED CAUSES MEMORY INCREASE 
    ms := &MyStruct{}
    size := 1000000000
    ms.Init(size, nil, nil)
    */

    fmt.Printf("finished allocating..%d %d\n", len(ms.arr1), len(ms.arr2))

    fmt.Printf("reading from stdin...\n")
    reader := bufio.NewReader(os.Stdin)

    var line string
    var readErr error
    var lineNo int = 0
    for {
        if lineNo%1000000 == 0 {
            fmt.Printf("read %d lines...\n", lineNo)
        }
        lineNo++

        line, readErr = reader.ReadString('\n')
        if readErr != nil {
            fmt.Printf("break at %s\n", line)
            break
        }
    }

    if readErr == io.EOF {
        readErr = nil
    }

    if readErr != nil {
        return readErr
    }

    return nil
}

func main() {
    if err := doSomething(); err != nil {
        panic(err)
    }
    fmt.Printf("done...\n")
}

1. 这是我的代码的问题吗？还是 go 系统在做一些意外的事情？
2. 如果是后者，我该如何调试它？

为了更容易复制，这里有用于好的情况(上面代码的注释部分)和坏情况(未注释部分)的 pastebin 文件

wget http://pastebin.com/raw/QfG22xXk -O badcase.go
yes "1234567890" | go run badcase.go

wget http://pastebin.com/raw/G9xS2fKy -O goodcase.go
yes "1234567890" | go run goodcase.go

Answer 1

感谢 Volker 的上述评论。我想捕捉调试这个过程作为答案。

RES top/htop 只是在进程级别告诉您内存正在发生什么。 GODEBUG="gctrace=1"让您更深入地了解内存的处理方式。

使用 gctrace 集的简单运行给出以下结果

root@localhost ~ # yes "12345678901234567890123456789012" | GODEBUG="gctrace=1" go run badcase.go
starting...
allocating
initializing mystruct arr1...
initializing mystruct arr2...
gc 1 @0.050s 0%: 0.19+0.23+0.068 ms clock, 0.58+0.016/0.16/0.25+0.20 ms cpu, 7629->7629->7629 MB, 7630 MB goal, 8 P
done initializing ...
gc 2 @0.100s 0%: 0.070+2515+0.23 ms clock, 0.49+0.025/0.096/0.24+1.6finished allocating..1000000000 1000000000
 ms cpu, 15258->15258reading from stdin...
->15258 MB, 15259read 0 lines...
 MB goal, 8 P
gc 3 @2.620s 0%: 0.009+0.32+0.23 ms clock, 0.072+0/0.20/0.11+1.8 ms cpu, 15259->15259->15258 MB, 30517 MB goal, 8 P

read 1000000 lines...
read 2000000 lines...
read 3000000 lines...
read 4000000 lines...
....
read 51000000 lines...
read 52000000 lines...
read 53000000 lines...
read 54000000 lines...

What does this mean ?

如您所见，gc 已经有一段时间没有被调用了。这意味着 reader.ReadString 生成的所有垃圾都没有被收集和释放。

Why isn't the garbage collector collecting this garbage ?

来自 The go gc

Instead we provide a single knob, called GOGC. This value controls the total size of the heap relative to the size of reachable objects. The default value of 100 means that total heap size is now 100% bigger than (i.e., twice) the size of the reachable objects after the last collection.

由于未设置 GOGC - 默认值为 100%。因此，它只有在达到 ~32GB 时才会收集垃圾。 (因为最初这两个数组为您提供了 16GB 的堆空间 - 只有当堆空间翻倍时才会触发 gc)。

How can I change this ? Try setting the GOGC=25.

以 GOGC 为 25

root@localhost ~ # yes "12345678901234567890123456789012" | GODEBUG="gctrace=1" GOGC=25 go run badcase.go
starting...
allocating
initializing mystruct arr1...
initializing mystruct arr2...
gc 1 @0.051s 0%: 0.14+0.30+0.11 ms clock, 0.42+0.016/0.31/0.094+0.35 ms cpu, 7629->7629->7629 MB, 7630 MB goal, 8 P
done initializing ...
finished allocating..1000000000 1000000000
gc 2 @0.102s reading from stdin...
12%: 0.058+2480+0.26 ms clock, 0.40+0.022/2480/0.10+1.8 ms cpu, 15258->15258->15258 MB, 15259 MB goal, 8 P
read 0 lines...
gc 3 @2.584s 12%: 0.009+0.20+0.22 ms clock, 0.075+0/0.24/0.046+1.8 ms cpu, 15259->15259->15258 MB, 19073 MB goal, 8 P
read 1000000 lines...
read 2000000 lines...
read 3000000 lines...
read 4000000 lines...
....
read 19000000 lines...
read 20000000 lines...
gc 4 @6.539s 4%: 0.019+2.3+0.23 ms clock, 0.15+0/2.1/12+1.8 ms cpu, 17166->17166->15258 MB, 19073 MB goal, 8 P

如你所见，触发了另一个 gc。

But top/htop shows it stable at ~20 GB instead of the calculated 16GB.

垃圾收集器并不“必须”将其还给操作系统。它有时会保留它以备将来有效使用。它不必继续从操作系统获取并返回 - 在再次询问操作系统之前，额外的 4 GB 在其可用空间池中使用。