Docker 统计数据显示内存使用量少于 top 命令的输出

Question

我在 docker 容器 app 中运行我的服务 app-apitop 的结果:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
6420 root      20   0 30.572g 0.028t  38956 S  47.8 92.5 240:40.95 app
...

htop 的结果:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 6420 root       20   0 30.6G 29.0G 38956 S 47.1 92.5  4h21:53 app
 6554 root       20   0 30.6G 29.0G 38956 S  6.6 92.5 23:04.15 app
 6463 root       20   0 30.6G 29.0G 38956 S  2.0 92.5 27:29.53 app
 6430 root       20   0 30.6G 29.0G 38956 S  0.0 92.5 25:30.61 app
 6429 root       20   0 30.6G 29.0G 38956 S  5.3 92.5 26:36.17 app
 6428 root       20   0 30.6G 29.0G 38956 S 10.0 92.5 23:56.10 app
 6426 root       20   0 30.6G 29.0G 38956 S  6.0 92.5  8:09.12 app
 6427 root       20   0 30.6G 29.0G 38956 S  0.0 92.5 23:03.81 app
 6425 root       20   0 30.6G 29.0G 38956 S  0.0 92.5  0:00.00 app
 6424 root       20   0 30.6G 29.0G 38956 S  0.0 92.5 25:42.46 app
 6423 root       20   0 30.6G 29.0G 38956 S  4.6 92.5 26:10.82 app
 6422 root       20   0 30.6G 29.0G 38956 S 12.0 92.5 23:24.68 app
 6421 root       20   0 30.6G 29.0G 38956 S  2.0 92.5  4:32.47 app
 2202 gitlab-ru  20   0  231M 70132 53620 S  5.3  0.2  4h54:21 nginx: worker process
 2203 gitlab-ru  20   0  228M 59040 47680 S  0.7  0.2 54:44.83 nginx: worker process
  281 root       19  -1  175M 58104 47728 S  0.0  0.2  0:17.76 /lib/systemd/systemd-journald
 1036 root       20   0 1893M 38164 13332 S  0.0  0.1  0:38.17 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
...

docker stats 的结果:

$ docker stats --no-stream
CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
14654b8a4bfb   app-letsencrypt  13.08%    244.5MiB / 31.41GiB   0.76%     183GB / 192GB     12.4GB / 4.64MB   23
a932dabbced8   app-api          60.50%    7.258GiB / 31.41GiB   23.10%    53.2GB / 10.6GB   48.1MB / 0B       14
2cebc542dda6   app-redis        0.12%     3.902MiB / 31.41GiB   0.01%     24.2kB / 0B       1.84GB / 655kB    4

正如您所看到的，top 中的 0.028t(~29G)远远超过 docker stats 中的 7.258GiB。差异大约是 29 - 7.258 > 20G 的 RAM。
请帮助我了解如何检测这个占用 20G RAM 的幻象是什么？或者也许可以指出我在哪里可以挖掘我的应用程序或docker(20.10.1)或操作系统(Ubuntu 18.04)的问题？
UPD
pprof 中的输出(堆)

# runtime.MemStats
# Alloc = 7645359160
# TotalAlloc = 2552206192400
# Sys = 31227357832
# Lookups = 0
# Mallocs = 50990505448
# Frees = 50882282691
# HeapAlloc = 7645359160
# HeapSys = 29526425600
# HeapIdle = 21707890688
# HeapInuse = 7818534912
# HeapReleased = 9017090048
# HeapObjects = 108222757
# Stack = 1474560 / 1474560
# MSpan = 101848496 / 367820800
# MCache = 13888 / 16384
# BuckHashSys = 10697838
# GCSys = 1270984696
# OtherSys = 49937954
# NextGC = 11845576832
# LastGC = 1615583458331895138
# PauseNs = ..................
# NumGC = 839
# NumForcedGC = 0
# GCCPUFraction = 0.027290987331299785
# DebugGC = false
# MaxRSS = 31197982720

Answer 1

您正在比较 top/htop RES mem ( man ):

The non-swapped physical memory a task has used. RES = CODE + DATA.

使用 docker stats CLI 输出 ( doc ):

On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage.

使用 docker stats API，您将获得更精细的 View ，例如内存统计:

{
    "total_pgmajfault": 0,
    "cache": 0,
    "mapped_file": 0,
    "total_inactive_file": 0,
    "pgpgout": 414,
    "rss": 6537216,
    "total_mapped_file": 0,
    "writeback": 0,
    "unevictable": 0,
    "pgpgin": 477,
    "total_unevictable": 0,
    "pgmajfault": 0,
    "total_rss": 6537216,
    "total_rss_huge": 6291456,
    "total_writeback": 0,
    "total_inactive_anon": 0,
    "rss_huge": 6291456,
    "hierarchical_memory_limit": 67108864,
    "total_pgfault": 964,
    "total_active_file": 0,
    "active_anon": 6537216,
    "total_active_anon": 6537216,
    "total_pgpgout": 414,
    "total_cache": 0,
    "inactive_anon": 0,
    "active_file": 0,
    "pgfault": 964,
    "inactive_file": 0,
    "total_pgpgin": 477
}

你可以看到 - memory 不只是一种，它有很多类型，每个工具可能会报告自己的内存类型集和组合。我猜您会在应用程序缓存内存分配中发现缺少内存。
您可以使用 free 命令检查总体基本内存分配:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           2000        1247          90         178         662         385
Swap:             0           0           0

当 Linux 将未使用的内存用于 buff/cache 时，这是一种正常状态。