python - 如何处理UserWarning : Converting sparse IndexedSlices to a dense Tensor of unknown shape

Question

我在 Tensorflow 中收到以下警告:UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape。这可能会消耗大量内存。

我得到这个的原因是:

import tensorflow as tf
# Flatten batch elements to rank-2 tensor where 1st max_length rows 
    #belong to first batch element and so forth
all_timesteps = tf.reshape(raw_output, [-1, n_dim])  # (batch_size*max_length, n_dim)
# Indices to last element of each sequence.
# Index to first element is the sequence order number times max 
    #sequence length.
# Index to last element is the index to first element plus sequence 
    #length.
row_inds = tf.range(0, batch_size) * max_length + (seq_len - 1)
# Gather rows with indices to last elements of sequences
# http://stackoverflow.com/questions/35892412/tensorflow-dense-gradient-explanation
# This is due to gather returning IndexedSlice which is later 
    #converted into a Tensor for gradient
# calculation.
last_timesteps = tf.gather(all_timesteps, row_inds)  # (batch_size,n_dim)  

tf.gather 导致了这个问题。直到现在我一直忽略它，因为我的架构并不是很大。但是，现在，我拥有更大的架构和大量数据。在使用大于 10 的批量大小进行训练时，我遇到了内存不足的问题。我相信处理此警告将使我能够将我的模型放入 GPU 中。

请注意，我使用的是 Tensorflow 1.3。

Answer 1

我设法通过使用 tf.dynnamic_partition 而不是 tf.gather 解决了这个问题。我将上面的代码替换成这样:

# Flatten batch elements to rank-2 tensor where 1st max_length rows belong to first batch element and so forth
all_timesteps = tf.reshape(raw_output, [-1, n_dim])  # (batch_size*max_length, n_dim)
# Indices to last element of each sequence.
# Index to first element is the sequence order number times max sequence length.
# Index to last element is the index to first element plus sequence length.
row_inds = tf.range(0, batch_size) * max_length + (seq_len - 1)
# Creating a vector of 0s and 1s that will specify what timesteps to choose.
partitions = tf.reduce_sum(tf.one_hot(row_inds, tf.shape(all_timesteps)[0], dtype='int32'), 0)
# Selecting the elements we want to choose.
last_timesteps = tf.dynamic_partition(all_timesteps, partitions, 2)  # (batch_size, n_dim)
last_timesteps = last_timesteps[1]