gpu - COS安装GPU下载驱动签名失败

标签 gpu google-compute-engine google-container-optimized-os

我在 COS 上使用了带有 T4 GPU 的 Compute Engine VM 很长一段时间，它一直运行良好，直到最近 cos-extensions install gpu 不像以前那样工作。

I0830 07:32:58.419130     987 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0830 07:32:58.427417     987 install.go:74] Running on COS build id 16108.470.16
I0830 07:32:58.427566     987 installer.go:187] Getting the default GPU driver version
I0830 07:32:58.427911     987 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548403     987 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548594     987 install.go:85] Installing GPU driver version 450.119.04
I0830 07:32:58.549646     987 cache.go:72] map[BUILD_ID:16108.470.11 DRIVER_VERSION:450.119.04]
I0830 07:32:58.549674     987 install.go:120] Did not find cached version, installing the drivers...
I0830 07:32:58.549681     987 installer.go:82] Configuring driver installation directories
I0830 07:32:58.563327     987 installer.go:196] Updating container's ld cache
I0830 07:32:58.793692     987 signature.go:30] Downloading driver signature for version 450.119.04
I0830 07:32:58.793721     987 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/16108.470.16/extensions/gpu/450.119.04.signature.tar.gz
E0830 07:32:58.828902     987 artifacts.go:106] Failed to download extensions/gpu/450.119.04.signature.tar.gz from public GCS: failed to download 450.119.04.signature.tar.gz, status: 404 Not Found
E0830 07:32:58.829401     987 install.go:175] failed to download driver signature: failed to download driver signature for version 450.119.04: failed to download extensions/gpu/450.119.04.signature.tar.gz

安装程序似乎找不到驱动程序签名。我调查了this并按照变通办法做

/usr/bin/docker run --rm \
    --privileged \
    --net=host \
    --pid=host \
    --volume /dev:/dev \
    --volume /:/root \
    --volume /var/lib/toolbox/nvidia:/usr/local/nvidia \
    --env NVIDIA_DRIVER_VERSION=450.119.04 \
    gcr.io/cos-cloud/cos-gpu-installer:latest

却得到了这个

+ COS_KERNEL_INFO_FILENAME=kernel_info
+ COS_KERNEL_SRC_HEADER=kernel-headers.tgz
+ TOOLCHAIN_URL_FILENAME=toolchain_url
+ TOOLCHAIN_ENV_FILENAME=toolchain_env
+ TOOLCHAIN_PKG_DIR=/build/cos-tools
+ CHROMIUMOS_SDK_GCS=https://storage.googleapis.com/chromiumos-sdk
+ ROOT_OS_RELEASE=/root/etc/os-release
+ KERNEL_SRC_HEADER=/build/usr/src/linux
+ NVIDIA_DRIVER_VERSION=450.119.04
+ NVIDIA_DRIVER_MD5SUM=
+ NVIDIA_INSTALL_DIR_HOST=/var/lib/nvidia
+ NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia
+ ROOT_MOUNT_DIR=/root
+ CACHE_FILE=/usr/local/nvidia/.cache
+ LOCK_FILE=/root/tmp/cos_gpu_installer_lock
+ LOCK_FILE_FD=20
+ set +x
[INFO    2021-08-30 07:36:38 UTC] PRELOAD: false
[INFO    2021-08-30 07:36:38 UTC] Running on COS build id 16108.470.16
[INFO    2021-08-30 07:36:38 UTC] Data dependencies (e.g. kernel source) will be fetched from https://storage.googleapis.com/cos-tools/16108.470.16
[INFO    2021-08-30 07:36:38 UTC] Checking if this is the only cos-gpu-installer that is running.
[INFO    2021-08-30 07:36:38 UTC] Checking if third party kernel modules can be installed
/tmp/esp /
/
[INFO    2021-08-30 07:36:38 UTC] Checking cached version
/entrypoint.sh: line 172: CACHE_BUILD_ID: unbound variable

似乎 COS 和 COS GPU 驱动程序正在发生一些变化(也许？)，但只想知道除了等待 GCP 解决问题之外是否有解决此问题的方法。

最佳答案

这与 Jan Vansteenlandt 的情况相同链接到。

在某些版本的 COS 中会出现这种情况；

例如现在可用的最新稳定 COS 版本 - 89-16108:

vm-16108 ~ # cos-extensions list Available extensions for COS version
89-16108.470.16:

[gpu]

[gpu] 下没有列出驱动程序正在运行 cos-extensions install gpu以与您的情况相同的方式结束。当尝试运行您提到的 docker 容器时，也产生了相同的结果。

这是一个已知问题，已在 IssueTracker 上提出。您可以关闭链接并单击 +1按钮，您也可以在话题中发表评论和发表您自己的发现。

线程中还有一个解决方法，因此您可以试一试。

如果您可以使用一些旧版本的 COS(例如 85-13310)- 列出了驱动程序:

vm-13310 ~ # cos-extensions list
Available extensions for COS version 85-13310.1308.10:

[gpu]
450.119.04 [default]

当你运行 cos-extensions install gpu它将导致成功安装 NVIDIA 驱动程序:


vm-13310 ~ # cos-extensions install gpu
I0831 14:25:11.405591    1168 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0831 14:25:11.407510    1168 install.go:74] Running on COS build id 13310.1308.10
I0831 14:25:11.407519    1168 installer.go:187] Getting the default GPU driver version
I0831 14:25:11.407581    1168 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448046    1168 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448539    1168 install.go:85] Installing GPU driver version 450.119.04
I0831 14:25:11.448751    1168 cache.go:69] error: failed to read file /root/var/lib/nvidia/.cache: open /root/var/lib/nvidia/.cache: no such file or directory
I0831 14:25:11.448942    1168 install.go:120] Did not find cached version, installing the drivers...
I0831 14:25:11.449084    1168 installer.go:82] Configuring driver installation directories
I0831 14:25:11.469718    1168 installer.go:196] Updating container's ld cache
I0831 14:25:11.480682    1168 signature.go:30] Downloading driver signature for version 450.119.04
I0831 14:25:11.481007    1168 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506186    1168 utils.go:120] Successfully downloaded 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506541    1168 signature.go:37] Decompressing signature /build/sign-gpu-driver/450.119.04.signature.tar.gz
I0831 14:25:11.510104    1168 installer.go:68] Downloading GPU driver installer version 450.119.04
I0831 14:25:11.511637    1168 utils.go:72] Downloading GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos
I0831 14:25:12.885856    1168 utils.go:120] Successfully downloaded GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos

-----  removed some lines for better readibility  -----

I0831 14:28:49.433597    1168 cache.go:58] Updated cached version as
I0831 14:28:49.498379    1168 cache.go:60] BUILD_ID=13310.1308.10
I0831 14:28:49.498560    1168 cache.go:60] DRIVER_VERSION=450.119.04
I0831 14:28:49.498694    1168 installer.go:32] Verifying GPU driver installation
I0831 14:28:50.309502    1168 utils.go:334] Tue Aug 31 14:28:50 2021       
I0831 14:28:50.309879    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.311093    1168 utils.go:334] | NVIDIA-SMI 450.119.04   Driver Version: 450.119.04   CUDA Version: 11.0     |
I0831 14:28:50.311300    1168 utils.go:334] |-------------------------------+----------------------+----------------------+
I0831 14:28:50.311497    1168 utils.go:334] | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
I0831 14:28:50.311640    1168 utils.go:334] | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
I0831 14:28:50.311784    1168 utils.go:334] |                               |                      |               MIG M. |
I0831 14:28:50.311949    1168 utils.go:334] |===============================+======================+======================|
I0831 14:28:50.322257    1168 utils.go:334] |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
I0831 14:28:50.322566    1168 utils.go:334] | N/A   76C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
I0831 14:28:50.322708    1168 utils.go:334] |                               |                      |                  N/A |
I0831 14:28:50.322878    1168 utils.go:334] +-------------------------------+----------------------+----------------------+
I0831 14:28:50.323119    1168 utils.go:334]                                                                                
I0831 14:28:50.323293    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.323431    1168 utils.go:334] | Processes:                                                                  |
I0831 14:28:50.323597    1168 utils.go:334] |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
I0831 14:28:50.323715    1168 utils.go:334] |        ID   ID                                                   Usage      |
I0831 14:28:50.323863    1168 utils.go:334] |=============================================================================|
I0831 14:28:50.324222    1168 utils.go:334] |  No running processes found                                                 |
I0831 14:28:50.324439    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.465730    1168 modules.go:48] Updating host's ld cache
I0831 14:28:52.305122    1168 install.go:167] Finished installing the drivers.

关于gpu - COS安装GPU下载驱动签名失败，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/68980540/

上一篇：azure - 为 VMSS 添加 Azure 诊断设置

下一篇：替换同维矩阵列表中的 for 循环以提高效率

相关文章：

c# - OpenCL 和 GPU 编程路线图

google-cloud-platform - GCP 通过 Compute Engine 的服务帐号访问和存储密码/凭据的正确方法

google-cloud-platform - 如何监控在 GCE 上运行的容器虚拟机的内存使用情况

google-cloud-platform - 如何从 GCE 实例中的 Container-optimized OS 获取启动脚本日志？

c++ - 是否可以为不同的 block 静态分配不同的共享内存？

tensorflow - GPU -> CPU Memcpy 在 tensorflow word2vec 中失败 gpu 发生

c# - 使用 GPU(可能是 CIFilters)将图像放置在更大的 Canvas 尺寸上而不使用图像上下文

metadata - 实例环境变量

google-compute-engine - 检测谷歌计算实例启动脚本完成了吗？

©2024 IT工具网  联系我们