python - 评估pytorch模型: `with torch.no_grad` vs `model.eval()`

Question

当我想评估模型在验证集上的性能时，是否首选使用 with torch.no_grad: 还是 model.eval()？

Answer 1

TL；博士:

Use both 。他们做不同的事情，并且有不同的范围。

• with torch.no_grad - 禁用 autograd 中的渐变跟踪。
• model.eval() 更改调用它的模块的 forward() 行为
• 例如，它禁用 dropout 并使用整个总体统计数据进行批量归一化

<小时/>

使用 torch.no_grad

torch.autograd.no_grad documentation说:

Context-manager that disabled [sic] gradient calculation.

Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.

model.eval()

nn.Module.eval documentation说:

Sets the module in evaluation mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

<小时/>

The creator of pytorch said the documentation should be updated to suggest the usage of both ，我提出了 pull request .