image - 在 Keras 中检查验证结果显示只有 50% 是正确的。明显随机

Question

我正在努力解决一个看似简单的问题。我不知道如何将我的输入图像与我的模型生成的结果概率相匹配。

我的模型(Vanilla VGG16，重新训练了 2 个类，狗和猫)的训练和验证进展顺利，使我的验证准确率接近 97%，但是当我运行检查时看看我做对了什么，做错了什么，我只能得到随机结果。

找到 1087 个正确的标签 (53.08%)

我很确定它与 ImageDataGenerator 有关，它在我的验证图像上生成随机批处理，尽管我确实设置了 shuffle = false

我只是在运行生成器之前保存它们的文件名和类，并且我假设我的文件名和类的索引与我的概率输出相同。

这是我的设置(Vanilla VGG16，替换了最后一层以匹配猫和狗的 2 个类别)

new_model.summary()

---

Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 224, 224, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
Binary_predictions (Dense)   (None, 2)                 8194      
=================================================================
Total params: 134,268,738
Trainable params: 8,194
Non-trainable params: 134,260,544
_________________________________________________________________

batch_size=16
epochs=3
learning_rate=0.01

这是生成器的定义，用于训练和验证。此时我还没有包括数据扩充部分。

train_datagen = ImageDataGenerator()
validation_datagen = ImageDataGenerator()
test_datagen = ImageDataGenerator()

train_generator = train_datagen.flow_from_directory(
    train_path,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='categorical')
train_filenames = train_generator.filenames
train_samples = len(train_filenames)

validation_generator = validation_datagen.flow_from_directory(
    valid_path,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='categorical',
    shuffle = False) #Need this to be false, so I can extract the correct classes and filenames in order that that are predicted
validation_filenames = validation_generator.filenames
validation_samples = len(validation_filenames)

微调模型没问题

#Fine-tune the model
#DOC: fit_generator(generator, steps_per_epoch, epochs=1, verbose=1, callbacks=None,
#              validation_data=None, validation_steps=None, class_weight=None,
#              max_queue_size=10, workers=1, use_multiprocessing=False, initial_epoch=0)

new_model.fit_generator(
    train_generator,
    steps_per_epoch=train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=validation_samples // batch_size)

Epoch 1/3
1434/1434 [==============================] - 146s - loss: 0.5456 - acc: 0.9653 - val_loss: 0.5043 - val_acc: 0.9678
Epoch 2/3
1434/1434 [==============================] - 148s - loss: 0.5312 - acc: 0.9665 - val_loss: 0.4293 - val_acc: 0.9722
Epoch 3/3
1434/1434 [==============================] - 148s - loss: 0.5332 - acc: 0.9665 - val_loss: 0.4329 - val_acc: 0.9731

验证数据的提取

#We need the probabilities/scores for the validation set
#DOC: predict_generator(generator, steps, max_queue_size=10, workers=1,
#                       use_multiprocessing=False, verbose=0)
probs = new_model.predict_generator(
            validation_generator,
            steps=validation_samples // batch_size,
            verbose = 1)

#Extracting the probabilities and labels
our_predictions = probs[:,0]
our_labels = np.round(1-our_predictions)
expected_labels = validation_generator.classes

现在，当我通过比较预期标签和计算标签来计算我的验证集是否成功时，我得到了一些非常接近随机的东西:

correct = np.where(our_labels==expected_labels)[0]
print("Found {:3d} correct labels ({:.2f}%)".format(len(correct),
       100*len(correct)/len(our_predictions)))

找到 1087 个正确的标签 (53.08%)

显然这是不正确的。

我怀疑这与生成器的随机性有关，但我设置了 shuffle = False。

这段代码是从伟大的 Jeremy Howard 的 Fast.ai 类(class)中直接复制的，但我无法让它继续工作了..

我在 Anaconda 下的 Python 3.5 上使用 Keras 2.0.8 和 TensorFlow 1.3 后端...

请帮助我保持理智!

Answer 1

您需要在 fit_generator() 和 predict_generator() 之间调用 validation_generator.reset()。

在 *_generator() 函数中，数据批处理在用于拟合/评估模型之前被插入到队列中。底层队列始终保持满状态，因此训练结束时队列中会有一些额外的批处理。您可以在训练后通过打印 validation_generator.batch_index 来验证它。因此，你的predict_generator()并不是从第一批开始的，probs[0]也不是第一张图片的预测。这就是为什么 our_labels 与 expected_labels 不对齐并且准确度低的原因。

顺便说一句，您应该使用 validation_steps=validation_samples//batch_size + 1(也用于训练生成器)。除非 validation_samples 是 batch_size 的倍数，否则如果您使用 validation_steps=validation_samples//batch_size，您将在每个时期忽略一个批处理，并且您的模型在每个时期的(略有)不同的数据集上进行评估。