python - 如何更改 Keras 中 softmax 输出的温度

Question

我目前正在尝试重现以下文章的结果。
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
我在 theano 后端使用 Keras。在文章中，他谈到了控制最终 softmax 层的温度以提供不同的输出。

Temperature. We can also play with the temperature of the Softmax during sampling. Decreasing the temperature from 1 to some lower number (e.g. 0.5) makes the RNN more confident, but also more conservative in its samples. Conversely, higher temperatures will give more diversity but at cost of more mistakes (e.g. spelling mistakes, etc). In particular, setting temperature very near zero will give the most likely thing that Paul Graham might say:

我的模型如下。

model = Sequential()
model.add(LSTM(128, batch_input_shape = (batch_size, 1, 256), stateful = True, return_sequences = True))
model.add(LSTM(128, stateful = True))
model.add(Dropout(0.1))
model.add(Dense(256, activation = 'softmax'))

model.compile(optimizer = Adam(),
              loss = 'categorical_crossentropy', 
              metrics = ['accuracy'])

我认为调整最终密集层温度的唯一方法是获取权重矩阵并将其乘以温度。有谁知道更好的方法吗？另外，如果有人发现我设置模型的方式有任何问题，请告诉我，因为我是 RNN 的新手。

Answer 1

看起来温度是您对 softmax 层的输出所做的事情。我找到了这个例子。

https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py

他应用以下函数对 soft-max 输出进行采样。

def sample(a, temperature=1.0):
    # helper function to sample an index from a probability array
    a = np.log(a) / temperature
    a = np.exp(a) / np.sum(np.exp(a))
    return np.argmax(np.random.multinomial(1, a, 1))