haskell - 反向传播算法可以改变权重的符号吗？

Question

我有一个业余项目，涉及使用动态数据集训练神经网络。我认为我已经正确实现了它，对于一些起始网络，我可以训练它们以匹配样本数据。我注意到训练时正权重不会变成负权重，反之亦然。这引出了两个问题:

1. “0 处的障碍”是否限制给定起始的行为 网络可以学习吗？
2. 如果是:我如何修改增量规则以便 权重可以在正数和负数之间变化吗？

这是我目前正在使用的代码(在 Haskell 中)。其中可能存在对其他人来说显而易见的错误，但它基本上是有效的。

import Data.List
import System.Random

type Layer = [[Double]]
type NNet = [Layer]

sigmoid :: Double -> Double
sigmoid t = 1 / (1 + exp (-t))

-- This is the derivative given the sigmoid value.
-- For the derivative given x: use sigmoidDerivative . sigmoid
sigmoidDerivative :: Double -> Double
sigmoidDerivative fx = fx * (1 - fx)

feedforward = flip (foldl' feedlayer)

feedlayer i = map (sigmoid . sum . zipWith (*) (1:i))

backprop :: Double -> [Double] -> [Double] -> NNet -> NNet
backprop rate i t n = fst $ backprop' i t n where
  backprop' i t (l:n) = (nw:r,be) where
    hs = feedlayer i l
    (r,e) = case n of
      [] -> ([], zipWith subtract hs t)
      x -> backprop' hs t n
    we = zipWith (\oe w ->
      map (*oe) w
     ) e l
    nw = zipWith3 (\wl dl h -> let sdh = sigmoidDerivative h in
      zipWith3 (\w d x ->
        w + rate * d * sdh * x
       ) wl dl (1:i)
     ) l we hs
    be = map sum $ transpose we

randomNNet :: RandomGen g => g -> [Int] -> NNet
randomNNet _ [_] = []
randomNNet gen (i:r@(n:_)) = let
  frl g _ = mapAccumL (\g _ -> let
    (a,g') = randomR (-0.05,0.05) g
    in (g',a)) g $ replicate (i+1) ()
  frg g = mapAccumL frl g $ replicate n ()
  (gen',l1) = frg gen
  in l1 : randomNNet gen' r

Answer 1

我犯了一个错误

当错误信号到达节点时，应该发生以下情况:

      -->Multiply by weights-->Propagate to earlier nodes
     /
error
     \
      -->Update weights.

我上面发布的代码中发生的事情是这样的:

                            -->Propagate to earlier nodes
                           /
error-->Multiply by weights
                           \
                            -->Update weights

这是固定代码，包括我插入的注释，以帮助自己弄清楚发生了什么。

backprop :: Double -> [Double] -> [Double] -> NNet -> NNet
backprop rate i t n = fst $ backprop' i t n where
  backprop' i t (l:n) = (nw:r,be) where
    -- hs: output of this layer
    hs = feedlayer i l
    -- r: the next layer updated
    -- e: the error of this layer's output
    (r,e) = case n of
      [] -> ([], zipWith subtract hs t)
      x -> backprop' hs t n
    -- we: Error divided among weights
    we = zipWith (\oe w ->
      map (*oe) w
     ) e l
    -- nw: New weights for this layer
    -- wl: weights leading to current node
    -- h: this node's output
    nw = zipWith3 (\wl d h -> let sdh = sigmoidDerivative h in
      -- w: The current weight
      -- d: The error assigned to the current node
      -- x: The input to the current synapse
      zipWith (\w x ->
        w + rate * d * sdh * x
       ) wl (1:i)
     ) l e hs
    -- be: Errors to propagate back to earlier nodes
    be = map sum $ transpose we