python - CUDNN_STATUS_ALLOC_FAILED 导致 Tensorflow 崩溃

Question

我在网上搜索了几个小时没有结果，所以想在这里问一下。

我正尝试按照 Sentdex 的教程制作一辆自动驾驶汽车，但在运行模型时，我遇到了一堆 fatal error 。我在网上搜索了解决方案，很多人似乎都有同样的问题。但是，我找到的所有解决方案(包括 this Stack-post)都不适合我。

这是我的软件:

• Tensorflow:1.5，GPU 版本
• CUDA:9.0，有补丁
• CUDnn:7
• Windows 10 专业版
• python 3.6

硬件:

• Nvidia 1070ti，带最新驱动
• 英特尔 i5 7600K

这是崩溃日志:

2018-02-04 16:29:33.606903: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.608872: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.609308: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145249: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145563: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM 2018-02-04 16:29:35.149896: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

这是我的代码:

 import tensorflow as tf
    import numpy as np
    import cv2
    import time
    from PIL import ImageGrab
    from getkeys import key_check
    from alexnet import alexnet
    import os
    from sendKeys import PressKey, ReleaseKey, W,A,S,D,Sp

    import random

    WIDTH = 80
    HEIGHT = 60
    LR = 1e-3
    EPOCHS = 10
    MODEL_NAME = 'DiRT-AI-Driver-{}-{}-{}-epochs.model'.format(LR, 'alexnetv2', EPOCHS)

    def straight():
        PressKey(W)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def left():
        PressKey(A)
        ReleaseKey(W)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def right():
        PressKey(D)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(W)
        ReleaseKey(Sp)
    def brake():
        PressKey(S)
        ReleaseKey(A)
        ReleaseKey(W)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def handbrake():
        PressKey(Sp)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(W)

    model = alexnet(WIDTH, HEIGHT, LR)
    model.load(MODEL_NAME)


    def main():
        last_time = time.time()
        for i in list(range(4))[::-1]:
            print(i+1)
            time.sleep(1)


    paused = False
    while(True):
            if not paused:
                screen = np.array(ImageGrab.grab(bbox=(0,40,1024,768)))
                screen = cv2.cvtColor(screen,cv2.COLOR_BGR2GRAY)
                screen = cv2.resize(screen,(80,60))
                print('Loop took {} seconds'.format(time.time()-last_time))
                last_time = time.time()
                print('took time')
                prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0]
                print('predicted')
                moves = list(np.around(prediction))
                print('got moves')
                print(moves,prediction)

                if moves == [1,0,0,0,0]:
                    straight()
                elif moves == [0,1,0,0,0]:
                    left()
                elif moves == [0,0,1,0,0]:
                    brake()
                elif moves == [0,0,0,1,0]:
                    right()
                elif moves == [0,0,0,0,1]:
                    handbrake()

            keys = key_check()

            if 'T' in keys:
                if paused:
                    pased = False
                    time.sleep(1)
                else:
                    paused = True
                    ReleaseKey(W)
                    ReleaseKey(A)
                    ReleaseKey(S)
                    ReleaseKey(D)
                    ReleaseKey(Sp)
                    time.sleep(1)


main()

我发现导致 python 崩溃并产生前三个错误的行是这一行:

• prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0]

运行代码时，CPU 占用率高达 100%，表明出现严重问题。 GPU 达到约 40-50%

我已经尝试过 Tensorflow 1.2 和 1.3，以及 CUDA 8，但都没有用。安装 CUDA 时，我没有安装特定的驱动程序，因为它们对我的 GPU 来说太旧了。也尝试了不同的 CUDnn，但效果不佳。

Answer 1

在我的例子中，问题的发生是因为另一个导入了 tensorflow 的 python 控制台正在运行。关闭它解决了问题。

我有 Windows 10，主要错误是:

failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED