python - 堆叠 LSTM 单元所有输出的总和 - Tensorflow

Question

在这里您可能会看到 TensorFlow 中多个堆叠 LSTM 单元的标准实现

with tf.name_scope("RNN_layers"):
    def lstm_cell():
        lstm = tf.contrib.rnn.LayerNormBasicLSTMCell(lstm_size)
        return lstm
    cell = tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(num_layers)])

with tf.name_scope("RNN_init_state"):
    initial_state = cell.zero_state(batch_size, tf.float32)

with tf.name_scope("RNN_forward"):
    outputs, state = tf.nn.dynamic_rnn(cell, inputs, initial_state=initial_state)

这对于许多任务来说非常有效。然而，在其他方面，一些专家建议将堆中单元格的所有输出的总和作为最终输出，沿着num_layers方向，而不仅仅是最后一个单元格的输出。

在下图中，要求是 y_t=h_t^1+h_t^2+h_t^3

在 TensorFlow 中实现此功能最明智的方法是什么？

Answer 1

从 tf.nn.dynamic_rnn 获得的 outputs 张量是所有单元的输出列表。如果你想计算它们的总和，只需调用 tf.reduce_sum关于输出:

n_steps = 2
n_inputs = 3
n_neurons = 5
X = tf.placeholder(dtype=tf.float32, shape=[None, n_steps, n_inputs])

basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
# outputs = [?, n_steps, n_neurons], e.g. outputs from all cells

sum = tf.reduce_sum(outputs, axis=1)
# sum = [?, n_neurons]

对于MultiRNNCell，它将是最后一层的输出之和，这也是您通常想要的。

---

更新:

对隐藏层的张量求和会更加困难，因为tensorflow MultiRNNCell为每个单元的输出重用相同的张量，因此隐藏层永远不会暴露出来RNN 的。

最简单的解决方案是编写自己的 MultiRNNCell这将总结每一层的输出，而不是只记住最后一层。您可以按以下方式执行此操作:

from tensorflow.python.util import nest

class MyMultiRNNCell(tf.nn.rnn_cell.MultiRNNCell):
  def call(self, inputs, state):
    cur_state_pos = 0
    cur_inp = inputs
    new_states = []
    new_outputs = []
    for i, cell in enumerate(self._cells):
      with tf.variable_scope("cell_%d" % i):
        if self._state_is_tuple:
          if not nest.is_sequence(state):
            raise ValueError("Expected state to be a tuple of length %d, but received: %s" %
                (len(self.state_size), state))
          cur_state = state[i]
        else:
          cur_state = tf.slice(state, [0, cur_state_pos], [-1, cell.state_size])
          cur_state_pos += cell.state_size
        cur_inp, new_state = cell(cur_inp, cur_state)
        new_states.append(new_state)
        new_outputs.append(cur_inp)

    new_states = (tuple(new_states) if self._state_is_tuple else
                  tf.concat(new_states, 1))
    new_outputs_sum = tf.reduce_sum(new_outputs, axis=0)

    return new_outputs_sum, new_states