docker - 在撰写文件中配置了GPU的Docker服务; Keras无法识别GPU

Question

我在v3.5 docker compose文件中配置了一个多服务应用程序。

其中一项服务是可以访问群集中(一个)节点上的(一个)GPU。
但是，如果我使用docker compose文件启动服务，那么我就无法访问GPU，如keras所报告:

   import keras
   from tensorflow.python.client import device_lib
   print(device_lib.list_local_devices())

版画

Using TensorFlow backend.

[name: "/device:CPU:0"
 device_type: "CPU"
 memory_limit: 268435456
 locality {  }
 incarnation: 10790773049987428954,
 name: "/device:XLA_CPU:0"
 device_type: "XLA_CPU"
 memory_limit: 17179869184
 locality {  }
 incarnation: 239154712796449863
 physical_device_desc: "device: XLA_CPU device"]

如果我从命令行像这样运行相同的图像:
docker run -it --rm $(ls /dev/nvidia* | xargs -I{} echo '--device={}') $(ls /usr/lib/*-linux-gnu/{libcuda,libnvidia}* | xargs -I{} echo '-v {}:{}:ro') -v $(pwd):/srv --entrypoint /bin/bash ${MY_IMG}
输出是

[name: "/device:CPU:0"
 device_type: "CPU"
 memory_limit: 268435456
 locality {
 }
 incarnation: 3178082198631681841, name: "/device:XLA_CPU:0"
 device_type: "XLA_CPU"
 memory_limit: 17179869184
 locality {
 }
 incarnation: 15685155444461741733
 physical_device_desc: "device: XLA_CPU device", name: "/device:XLA_GPU:0"
 device_type: "XLA_GPU"
 memory_limit: 17179869184
 locality {
 }
 incarnation: 4056441191345727860
 physical_device_desc: "device: XLA_GPU device"]

配置:

我已经安装了nvidia-docker并根据this guide配置了节点:

/etc/systemd/system/docker.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-runtime=nvidia --node-generic-resource gpu=GPU-b7ad85d5

和

/etc/nvidia-container-runtime/config.toml:

disable-require = false
swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"

docker 组成文件的相关部分:

docker-compose.yaml:

version: '3.5'
   ...
   services:
        ...
        my-service:
            ...
            deploy:
              resources:
                reservations:
                  generic_resources:
                    - discrete_resource_spec:
                        kind: 'gpu'
                        value: 1

问题:
要访问该docker服务中的GPU，还需要什么？

Answer 1

NVIDIA-Docker仅在Docker Compose 2.3上运行

将版本更改为version: '2.3'
https://github.com/NVIDIA/nvidia-docker/wiki#do-you-support-docker-compose。