Machine Learning
Activation Functions
The nonlinearity between layers is what lets a network bend, not just scale. Compare the classic activations and, just as importantly, their derivatives: where the slope goes flat, gradients vanish and learning stalls.
f(x)—
f′(x) (gradient)—
What to observe
- Teal is the activation, amber is its derivative, the factor every gradient gets multiplied by during backprop. Where amber sits near zero, that layer barely learns.
- Drag x into the tails of Sigmoid or tanh: the curve flattens and f′ collapses toward 0. Stack many such layers and gradientsvanish, the problem ReLU was invented to dodge.
- ReLU keeps a slope of exactly 1 for x > 0 (no vanishing) but is flat and dead for x < 0. Leaky and GELU patch that dead zone in different ways.
Shortcuts: space run/pause · s step · r reset · f fullscreen