machine-learning - TensorFlow 中的批量标准化

Question

在 TensorFlow 中执行批量标准化的正确方法是什么？ (即我不想计算运行平均值和方差)。我当前的实现基于 tf.nn.batch_normalization ，其中 x 是形状为 [batch_size, width, height, num_channels] 的卷积层的输出。我想明智地执行批量归一化 channel 。

batch_mean, batch_var = tf.nn.moments(x, axes=[0, 1, 2])
x = tf.nn.batch_normalization(x, batch_mean, batch_var, offset=0, scale=0, variance_epsilon=1e-6)

但是这个实现的结果非常糟糕。与tensorflow.contrib.slim.batch_norm的比较表明它较差(同样糟糕的训练性能)。

我做错了什么，如何解释这种糟糕的表现？

Answer 1

您可以考虑tf.contrib.layers.layer_norm。您可能需要将 x reshape 为 [batch、channel、width、height] 并设置 begin_norm_axis=2 进行 channel 标准化(每个批处理和每个 channel 将独立标准化) .

以下示例如何将原始订单 reshape 为[批处理、 channel 、宽度、高度]:

import tensorflow as tf

sess = tf.InteractiveSession()

batch = 2
height = 2
width = 2
channel = 3

tot_size = batch * height * channel * width

ts_4D_bhwc = tf.reshape(tf.range(tot_size), [batch, height, width, channel])
ts_4D_bchw = tf.transpose(ts_4D_bhwc, perm=[0,3,1,2])

print("Original tensor w/ order bhwc\n")
print(ts_4D_bhwc.eval())

print("\nTransormed tensor w/ order bchw\n")
print(ts_4D_bchw.eval())

输出:

Original tensor w/ order bhwc

[[[[ 0  1  2]
   [ 3  4  5]]

  [[ 6  7  8]
   [ 9 10 11]]]


 [[[12 13 14]
   [15 16 17]]

  [[18 19 20]
   [21 22 23]]]]

Transormed tensor w/ order bchw

[[[[ 0  3]
   [ 6  9]]

  [[ 1  4]
   [ 7 10]]

  [[ 2  5]
   [ 8 11]]]


 [[[12 15]
   [18 21]]

  [[13 16]
   [19 22]]

  [[14 17]
   [20 23]]]]