Deep Learning
Backpropagation
One tiny network, one example, every number on screen. Push the input forward, send the error back down the same wires, then see why the activation's slope decides whether a gradient survives many layers.
values going forwardgradients coming backthe updatea negative weight
Wire thickness is the size of the weight · click a cell of the strip to jump to that step
activation φ(z)slope φ′(z)saturated, |φ′| < 0.1
Click the top plot to move z
Step—
—
View
Activation φ
Target y
passes done—
prediction a[2]—
loss—
Gradients none yet
∂L/∂W[1]11—
∂L/∂W[1]12—
∂L/∂b[1]1—
∂L/∂W[1]21—
∂L/∂W[1]22—
∂L/∂b[1]2—
∂L/∂W[2]1—
∂L/∂W[2]2—
∂L/∂b[2]—
Loss after each pass
What to observe
- Step through one pass. Forward (teal), every unit does multiply, add, squash. Backward (amber), every unit multiplies by its local slope φ′ and hands the result back down the same wires.
- Steps 7 and 9 are one formula: a weight's gradient is the delta at its head times the activation at its tail. A bias has no tail, so its gradient is the delta itself.
- With ReLU, push unit 1's z[1] below zero. Its slope is exactly 0, so its δ and both weights into it get no gradient: adead unit.
- In the Activation view the bottom plot multiplies φ′ over n layers. σ′ never exceeds 0.25, so ten σ layers cut the gradient at least a millionfold: it vanishes. tanh′ peaks at 1 and tanh is zero-centred, but it saturates in its tails too; ReLU passes exactly 1 for z > 0.
Shortcuts:space run/pause · s step · r reset · f fullscreen