python - 使用 Climin 在 Theano 中实现逻辑回归

Question

我想使用 Theano 在 MNIST 上实现小批量逻辑回归。我想尝试不同的优化器，所以我决定使用一个名为 climin 的库。 Climin 提供了几个函数，它们接收模型参数、损失和/或损失梯度以及数据流作为参数。由于theano函数需要编译，我设计了以下模型。

# loads the MNIST dataset
train_set_x, train_set_y = load_data(dataset)[0]

print('... building the model')

W = theano.shared(
        value = np.zeros((28 * 28, 10), dtype = theano.config.floatX),
        name = 'W',
        borrow = True
        ) # weights dimension 28 * 28, 10

b = theano.shared(
        value = np.zeros((10,), dtype = theano.config.floatX),
        name = 'b',
        borrow = True
        ) # biases

x = T.matrix('x') # data, represented as rasterized images dimension 28 * 28
y = T.ivector('y') # labes, represented as 1D vector of [int] labels dimension 10

p_y_given_x             =  T.nnet.softmax(T.dot(x, W) + b)
y_pred                  =  T.argmax(p_y_given_x, axis=1)

NLL = -T.sum(T.log(p_y_given_x)[T.arange(y.shape[0]), y])

# negative_log_likelihood =  -T.mean(T.log(y_pred)[T.arange(y.shape[0]), y])

loss = theano.function(inputs = [ x, y ], outputs = NLL )

g_W = theano.function(inputs = [ x, y ], outputs = T.grad(cost=loss, wrt=W))
g_b = theano.function(inputs = [ x, y ], outputs = T.grad(cost=loss, wrt=b))

def d_loss_wrt_pars(parameters, inputs, targets):
    # climin passes the parametes
    # as 1-D concatinated array to
    # this function so I need to
    # unpack the parameter array
    W = parameters[:28 * 28 * 10].reshape((28 * 28, 10))
    b = parameters[28 * 28 * 10:].reshape((1, 10))

    return np.concatinate([g_W(x, y).flatten(), g_b(x, y)])

wrt = np.empty(7850) # allocate space for the parameters (MNIST dimensionality 28 * 28 * 10)
cli.initialize.randomize_normal(wrt, 0, 1) # initialize the parameters with random numbers

if batch_size is None:
    args = itertools.repeat(([train_set_x, train_set_y], {}))
    batches_per_pass = 1
else:
    args = cli.util.iter_minibatches([train_set_x, train_set_y], batch_size, [0, 0])
    args = ((i, {}) for i in args)
    batches_per_pass = train_set_x.shape[0] / batch_size

if optimizer == 'gd':
    opt = cli.GradientDescent(wrt, d_loss_wrt_pars, step_rate=0.1, momentum=.95, args=args)
else:
    print('unknown optimizer')
    return 1

print('... training the model')

for info in opt:
    if info['n_iter'] >= n_epochs and (not done_looping):
        break

不幸的是，这只会产生:

Traceback (most recent call last):
File "logreg/logistic_sgd.py", line 160, in <module> sys.exit(sgd_optimization_mnist())
File "logreg/logistic_sgd.py", line 69, in sgd_optimization_mnist
g_W = theano.function(inputs = [ x, y ], outputs = T.grad(cost=loss, wrt=W))
File "/Users/romancpodolski/anaconda/lib/python2.7/site-packages/theano/gradient.py", line 430, in grad
if cost is not None and isinstance(cost.type, NullType):

属性错误:“函数”对象没有属性“类型”

有人知道如何实现这项工作并结合两个库吗？

Answer 1

您的代码将编译函数(而不是 Theano 表达式)传递给 T.grad。将 loss 替换为 NLL 就可以了

g_W = theano.function(inputs = [ x, y ], outputs = T.grad(cost=NLL, wrt=W)) 
g_b = theano.function(inputs = [ x, y ], outputs = T.grad(cost=NLL, wrt=b))

此外，您可能想尝试使用 Theano 支持库(例如 Lasagne )进行优化。