python - 使用 Scipy minimize 进行 Keras BFGS 训练

Question

我想使用 BFGS 训练在 Keras 中实现的前馈神经网络。为了看看它是否可以完成，我使用 scipy.optimize.minimize 实现了一个感知器，代码如下。

from __future__ import print_function
import numpy as np
from scipy.optimize import minimize
from keras.models import Sequential
from keras.layers.core import Dense

# Dummy training examples
X = np.array([[-1,2,-3,-1],[3,2,-1,-4]]).astype('float')
Y = np.array([[2],[-1]]).astype('float')

model = Sequential()
model.add(Dense(1, activation='sigmoid', input_dim=4))

def loss(W):
    weightsList = [np.zeros((4,1)), np.zeros(1)]
    for i in range(4):
        weightsList[0][i,0] = W[i]
    weightsList[1][0] = W[4]
    model.set_weights(weightsList)
    preds = model.predict(X)
    mse = np.sum(np.square(np.subtract(preds,Y)))/len(X[:,0])
    return mse

# Dummy first guess
V = [1.0, 2.0, 3.0, 4.0, 1.0]
res = minimize(loss, x0=V, method = 'BFGS', options={'disp':True})
print(res.x)

但是，这个输出显示损失函数没有优化:

Using Theano backend.
Using gpu device 0: GeForce GTX 960M (CNMeM is disabled, cuDNN not available)
Optimization terminated successfully.
         Current function value: 2.499770
         Iterations: 0
         Function evaluations: 7
         Gradient evaluations: 1
[ 1.  2.  3.  4.  1.]

知道为什么这行不通吗？是不是因为minimize我没有输入梯度，所以在这种情况下无法计算数值近似值？

Answer 1

Is it because I didn't input the gradient to minimize, and it cannot calculate the numerical approximation in this case?

因为你没有输出梯度，所以scipy通过数值微分来近似它们。即它在 X 处评估函数，然后在 X + epsilon 处评估函数，以近似局部梯度。

但是 epsilon 足够小以至于在为 theano 转换为 32 位时，变化完全丢失了。开始猜测实际上不是最小值，scipy 只是这么认为，因为它认为目标函数的值没有变化。您只需要这样增加 epsilon:

V = [1.0, 2.0, 3.0, 4.0, 1.0]
print('Starting loss = {}'.format(loss(V)))
# set the eps option to increase the epsilon used in numerical diff
res = minimize(loss, x0=V, method = 'BFGS', options={'eps':1e-6,'disp':True})
print('Ending loss = {}'.format(loss(res.x)))

给出:

Using Theano backend.
Starting loss = 2.49976992001
Optimization terminated successfully.
         Current function value: 1.002703
         Iterations: 19
         Function evaluations: 511
         Gradient evaluations: 73
Ending loss = 1.00270344184