python - 线性回归和自动求导

Question

设 $F\in\mathbb{R}^{S\times F}$ 为特征矩阵，我想使用带有 autograd 的逻辑回归对它们进行分类 [1]。我使用的代码类似于以下示例 [2] 中的代码。

我唯一想要改变的是，我在 $\mathbb{R}^{F\times L}$ 中有一个额外的权重矩阵 $W$，我想将其应用于每个特征。因此，每个特征都与 $W$ 相乘，然后输入到逻辑回归中。

是否可以使用 autograd 同时训练 $W$ 和逻辑回归的权重？

我尝试了以下代码，不幸的是权重保持在值 0。

import autograd.numpy as np
from autograd import grad

    global inputs

    def sigmoid(x):
        return 0.5 * (np.tanh(x) + 1)


    def logistic_predictions(weights, inputs):
        # Outputs probability of a label being true according to logistic model.
        return sigmoid(np.dot(inputs, weights))


    def training_loss(weights):
        global inputs
        # Training loss is the negative log-likelihood of the training labels.

        feature_weights = weights[3:]
        feature_weights = np.reshape(feature_weights, (3, 3))

        inputs = np.dot(inputs, feature_weights)

        preds = logistic_predictions(weights[0:3], inputs)
        label_probabilities = preds * targets + (1 - preds) * (1 - targets)

        return -np.sum(np.log(label_probabilities))


    # Build a toy dataset.
    inputs = np.array([[0.52, 1.12, 0.77],
                       [0.88, -1.08, 0.15],
                       [0.52, 0.06, -1.30],
                       [0.74, -2.49, 1.39]])

    targets = np.array([True, True, False, True])

    # Define a function that returns gradients of training loss using autograd.
    training_gradient_fun = grad(training_loss)

    # Optimize weights using gradient descent.
    weights = np.zeros([3 + 3 * 3])
    print "Initial loss:", training_loss(weights)
    for i in xrange(100):
        print(i)
        print(weights)
        weights -= training_gradient_fun(weights) * 0.01

    print  "Trained loss:", training_loss(weights)

[1] https://github.com/HIPS/autograd

[2] https://github.com/HIPS/autograd/blob/master/examples/logistic_regression.py

Answer 1

典型的做法是将所有“向量化”参数连接到决策变量向量中。

如果您更新 logistic_predictions 以包含 W 矩阵，通过类似

def logistic_predictions(weights_and_W, inputs):
    '''
    Here, :arg weights_and_W: is an array of the form [weights W.ravel()]
    '''
    # Outputs probability of a label being true according to logistic model.
    weights = weights_and_W[:inputs.shape[1]]
    W_raveled = weights_and_W[inputs.shape[1]:]
    n_W = len(W_raveled)
    W = W_raveled.reshape(inputs.shape[1], n_W/inputs.shape[1])

    return sigmoid(np.dot(np.dot(inputs, W), weights))

然后只需将 traning_loss 更改为(来自原始源示例)

def training_loss(weights_and_W):
    # Training loss is the negative log-likelihood of the training labels.
    preds = logistic_predictions(weights_and_W, inputs)
    label_probabilities = preds * targets + (1 - preds) * (1 - targets)
    return -np.sum(np.log(label_probabilities))