matlab - 没有矩阵的多变量梯度下降

Question

我是 Matlab 和机器学习的新手，我尝试在不使用矩阵的情况下创建梯度下降函数。

• m 是我的训练集上的示例数量
• n 是每个示例的特征数量

函数gradientDescentMulti有5个参数:

• X mxn 矩阵
• y m 维向量
• theta:n维向量
• alpha:实数
• nb_iters:实数

我已经有了一个使用矩阵乘法的解决方案

function theta = gradientDescentMulti(X, y, theta, alpha, num_iters)
  for iter = 1:num_iters
    gradJ = 1/m * (X'*X*theta - X'*y);
    theta = theta - alpha * gradJ;
  end
end

迭代后的结果:

theta =
   1.0e+05 *

    3.3430
    1.0009
    0.0367

但是现在，我尝试在不进行矩阵乘法的情况下执行相同的操作，这是函数:

function theta = gradientDescentMulti(X, y, theta, alpha, num_iters)
  m = length(y); % number of training examples
  n = size(X, 2); % number of features

  for iter = 1:num_iters
    new_theta = zeros(1, n);
    %// for each feature, found the new theta
    for t = 1:n
      S = 0;
      for example = 1:m
        h = 0;
        for example_feature = 1:n
          h = h + (theta(example_feature) * X(example, example_feature));
        end
        S = S + ((h - y(example)) * X(example, n)); %// Sum each feature for this example
      end
      new_theta(t) = theta(t) - alpha * (1/m) * S; %// Calculate new theta for this example
    end 
    %// only at the end of the function, update all theta simultaneously
    theta = new_theta'; %// Transpose new_theta (horizontal vector) to theta (vertical vector)
  end
end

结果，所有的 theta 都相同:/

theta =
   1.0e+04 *

    3.5374
    3.5374
    3.5374

Answer 1

如果您查看梯度更新规则，首先实际计算所有训练示例的假设，然后用每个训练示例的真实值减去该假设并将其存储到数组或向量中可能会更有效。完成此操作后，您就可以非常轻松地计算更新规则。对我来说，您似乎没有在代码中执行此操作。

因此，我重写了代码，但我有一个单独的数组，用于存储每个训练示例的假设和真实值的差异。完成此操作后，我将分别计算每个功能的更新规则:

for iter = 1 : num_iters

    %// Compute hypothesis differences with ground truth first
    h = zeros(1, m);
    for t = 1 : m
        %// Compute hypothesis
        for tt = 1 : n
            h(t) = h(t) + theta(tt)*X(t,tt);
        end
        %// Compute difference between hypothesis and ground truth
        h(t) = h(t) - y(t);
    end

    %// Now update parameters
    new_theta = zeros(1, n);    
    %// for each feature, find the new theta
    for tt = 1 : n
        S = 0;
        %// For each sample, compute products of hypothesis difference
        %// and the right feature of the sample and accumulate
        for t = 1 : m
            S = S + h(t)*X(t,tt);
        end

        %// Compute gradient descent step
        new_theta(tt) = theta(tt) - (alpha/m)*S;
    end

    theta = new_theta'; %// Transpose new_theta (horizontal vector) to theta (vertical vector)    

end

当我这样做时，我得到了与使用矩阵公式相同的答案。