1.前向传播

将上一层的输出作为下一层的输入，并计算下一层的输出，一直到输出层为止

(图片源自网络)
使用矩阵表示

z^{(l)} = w^{(l)}a^{(l-1)} + b^{(l)}
a^{(l)} = \sigma(z^{(l)})

2.反向传播

记损失函数为L = C(W, b)

我们的目标是求\frac{\partial L}{\partial w^{(l)}} 和 \frac{\partial L}{\partial b^{(l)}}

根据链式法则，可以求出输出层的梯度

\frac{\partial L}{\partial w^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \frac{\partial a^{(l)}}{\partial z^{(l)}} \frac{\partial z^{(l)}}{\partial w^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \frac{\partial a^{(l)}}{\partial z^{(l)}} a^{(l-1)}

\frac{\partial L}{\partial b^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \frac{\partial a^{(l)}}{\partial z^{(l)}} \frac{\partial z^{(l)}}{\partial b^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \frac{\partial a^{(l)}}{\partial z^{(l)}}

记

\delta^{(l)} = \frac{\partial L}{\partial a^{(l)}} \frac{\partial a^{(l)}}{\partial z^{(l)}}

则隐藏层的梯度

\delta^{(l)} = \frac{\partial L}{\partial z^{(l+1)}} \frac{\partial z^{(l+1)}}{\partial z^{(l)}} = \delta^{(l+1)} \frac{\partial z^{(l+1)}}{\partial z^{(l)}}

又由

z^{(l+1)} = w^{(l+1)} \sigma(z^{(l)}) + b^{(l+1)}

得到

\frac{\partial z^{(l+1)}}{\partial z^{(l)}} = (w^{(l+1)})^T\odot\sigma(z^{(l)})

因此

\delta^{(l)} = \delta^{(l+1)} (w^{(l+1)})^T\odot\sigma(z^{(l)})

由\delta^{(l)}的递推关系式，可以很容易的求解w^{(l)} b^{(l)}的梯度

\frac{\partial L}{\partial w^{(l)}} = \delta^{(l)}a^{(l-1)}

\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}

下面是伪代码