我想在 TensorFlow 中使用 批量标准化。我在 core/ops/nn_ops.cc
中找到了相关的 C++ 源代码.但是,我没有在 tensorflow.org 上找到它。
BN 在 MLP 和 CNN 中有不同的语义,所以我不确定这个 BN 到底是做什么的。
我也没有找到名为 MovingMoments
的方法。
最佳答案
2016 年 7 月更新 在 TensorFlow 中使用批量标准化的最简单方法是通过 contrib/layers 中提供的高级接口(interface)。 , tflearn , 或 slim .
如果你想 DIY,上一个答案:
自发布以来,文档字符串已得到改进 - 请参阅 docs comment in the master branch而不是你找到的那个。它特别说明了它是 tf.nn.moments
的输出。
您可以在 batch_norm test code 中看到一个非常简单的使用示例。 .对于更真实的使用示例,我在帮助器类下方包含了我为自己使用而草草写下的使用说明(不提供任何保证!):
"""A helper class for managing batch normalization state.
This class is designed to simplify adding batch normalization
(http://arxiv.org/pdf/1502.03167v3.pdf) to your model by
managing the state variables associated with it.
Important use note: The function get_assigner() returns
an op that must be executed to save the updated state.
A suggested way to do this is to make execution of the
model optimizer force it, e.g., by:
update_assignments = tf.group(bn1.get_assigner(),
bn2.get_assigner())
with tf.control_dependencies([optimizer]):
optimizer = tf.group(update_assignments)
"""
import tensorflow as tf
class ConvolutionalBatchNormalizer(object):
"""Helper class that groups the normalization logic and variables.
Use:
ewma = tf.train.ExponentialMovingAverage(decay=0.99)
bn = ConvolutionalBatchNormalizer(depth, 0.001, ewma, True)
update_assignments = bn.get_assigner()
x = bn.normalize(y, train=training?)
(the output x will be batch-normalized).
"""
def __init__(self, depth, epsilon, ewma_trainer, scale_after_norm):
self.mean = tf.Variable(tf.constant(0.0, shape=[depth]),
trainable=False)
self.variance = tf.Variable(tf.constant(1.0, shape=[depth]),
trainable=False)
self.beta = tf.Variable(tf.constant(0.0, shape=[depth]))
self.gamma = tf.Variable(tf.constant(1.0, shape=[depth]))
self.ewma_trainer = ewma_trainer
self.epsilon = epsilon
self.scale_after_norm = scale_after_norm
def get_assigner(self):
"""Returns an EWMA apply op that must be invoked after optimization."""
return self.ewma_trainer.apply([self.mean, self.variance])
def normalize(self, x, train=True):
"""Returns a batch-normalized version of x."""
if train:
mean, variance = tf.nn.moments(x, [0, 1, 2])
assign_mean = self.mean.assign(mean)
assign_variance = self.variance.assign(variance)
with tf.control_dependencies([assign_mean, assign_variance]):
return tf.nn.batch_norm_with_global_normalization(
x, mean, variance, self.beta, self.gamma,
self.epsilon, self.scale_after_norm)
else:
mean = self.ewma_trainer.average(self.mean)
variance = self.ewma_trainer.average(self.variance)
local_beta = tf.identity(self.beta)
local_gamma = tf.identity(self.gamma)
return tf.nn.batch_norm_with_global_normalization(
x, mean, variance, local_beta, local_gamma,
self.epsilon, self.scale_after_norm)
请注意,我将其称为 ConvolutionalBatchNormalizer
,因为它使用 tf.nn.moments
对轴 0、1 和 2 求和,而对于非卷积使用你可能只需要轴 0。
如果您使用它,我们将不胜感激。
关于python - 如何在 TensorFlow 中使用批量标准化?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/33949786/