c# - Dotnet Core Docker 容器在 Linux 上泄漏 RAM 并导致 OOM

Question

我在 Docker 的 Linux 容器中运行 Dotnet Core 2.2。

我尝试了许多不同的配置/环境选项 - 但我不断回到内存不足的相同问题('docker events' 报告 OOM)。

在生产中，我在 Ubuntu 上托管。对于开发，我在 Windows 的 Docker 上使用 Linux 容器 (MobyLinux)。

我已经回到运行 Web API 模板项目，而不是我的实际应用程序。我实际上是在返回一个字符串并且什么都不做。如果我从 curl 调用它大约 1000 次，容器就会死掉。垃圾收集器似乎根本没有工作。

尝试在 docker-compose 中设置以下环境变量:

DOTNET_RUNNING_IN_CONTAINER=true
DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
ASPNETCORE_preventHostingStartup=true

还在 docker-compose 中尝试了以下操作:

mem_reservation: 128m
mem_limit: 256m
memswap_limit: 256m

(这些只会让它死得更快)

尝试将以下设置为真或假，没有区别:

ServerGarbageCollection

我尝试过作为 Windows 容器运行，这不会 OOM - 但它似乎也不尊重内存限制。

我已经排除了使用 HttpClient 和 EF Core - 因为我什至没有在我的示例中使用它们。我已经读过一些关于在端口 443 上监听的问题 - 因为我可以让容器整天闲置，如果我在一天结束时检查 - 它用掉了更多的内存(不是大量的，但它增长)。

我的 API 中的示例:

// GET api/values/5
[HttpGet("{id}")]
public ActionResult<string> Get(int id)
{
return "You said: " + id;
}

使用 Curl 调用示例:

curl -X GET "https://localhost:44329/api/values/7" -H  "accept: text/plain" --insecure

(重复1000次左右)

预期:对于一个非常原始的请求，RAM 使用率保持在较低水平

实际:RAM 使用量持续增长，直到出现故障

完整的 Dockerfile:

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /src
COPY ["WebApplication1/WebApplication1.csproj", "WebApplication1/"]
RUN dotnet restore "WebApplication1/WebApplication1.csproj"
COPY . .
WORKDIR "/src/WebApplication1"
RUN dotnet build "WebApplication1.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "WebApplication1.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "WebApplication1.dll"]

docker -compose.yml

version: '2.3'

services:
  webapplication1:
    image: ${DOCKER_REGISTRY-}webapplication1
    mem_reservation: 128m
    mem_limit: 256m
    memswap_limit: 256m
    cpu_percent: 25
    build:
      context: .
      dockerfile: WebApplication1/Dockerfile

docker-compose.override.yml

version: '2.3'

services:
  webapplication1:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=https://+:443;http://+:80
      - ASPNETCORE_HTTPS_PORT=44329
      - DOTNET_RUNNING_IN_CONTAINER=true
      - DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
      - ASPNETCORE_preventHostingStartup=true
    ports:
      - "50996:80"
      - "44329:443"
    volumes:
      - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
      - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro

我在 Windows 上运行 Docker CE Engine 18.0.9.1，在 Ubuntu 上运行 18.06.1。确认 - 我也尝试过 Dotnet Core 2.1。

我还在 IIS Express 中尝试过 - 进程达到大约 55MB，这实际上是用多个线程等向它发送垃圾邮件。

当它们都完成后，它会下降到大约 29-35MB。

Answer 1

这可能是因为没有执行垃圾回收 (GC)。
看看这个开放的问题，它看起来非常相似:
https://github.com/dotnet/runtime/issues/851
一种解决方案使 Ubuntu 18.04.4在虚拟机上工作时使用的是 Workstation 垃圾收集 (GC):

<PropertyGroup>
    <ServerGarbageCollection>false</ServerGarbageCollection>
</PropertyGroup>

https://github.com/dotnet/runtime/issues/851#issuecomment-644648315
https://github.com/dotnet/runtime/issues/851#issuecomment-438474207
https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc
这是另一个发现:

After further investigations I've noticed that there is big difference between my servers in amount of available logical CPUs count (80 vs 16). After some googling I came across this topic dotnet/runtime#622 that leads me to an experiments with CPU/GC/Threads settings.

I was using --cpus constraint in stack file; explicitly set System.GC.Concurrent=true, System.GC.HeapCount=8, System.GC.NoAffinitize=true, System.Threading.ThreadPool.MaxThreads=16 in runtimeconfig.template.json file; update image to a 3.1.301-bionic sdk and 3.1.5-bionic asp.net runtime — I made all this things in a various combinations and all of this had no effect. Application just hangs until gets OOMKilled.

The only thing that make it work with Server GC is --cpuset-cpus constraint. Of course, explicit setting of available processors is not an option for a docker swarm mode. But I was experimenting with available cpus to find any regularity. And here I got a few interesting facts.

What is interesting, previously I have mirgated 3 other backend services to a new servers cluster and they all go well with a default settings. Their memory limit is set to 600 Mb but in fact they need about 400 Mb to run. Things go wrong only with memory-consuming applications (I have two of those), it requires 3 Gb to build in-memory structures and runs with a 6 Gb constraint.

It keeps working in any range between [1, 35] available cpus and gets hanging when cpus count is 36.

https://github.com/dotnet/runtime/issues/851#issuecomment-645237830