Deep Learning
Initialisation & Normalisation
Every layer multiplies the spread of the signal by a factor, and depth compounds it. Pick a weight scale and watch the activations survive, vanish or overflow, then normalise each layer and see the scale stop compounding.
the signal going forwardgradients coming backthe other schemes, or no normalisation
The ribbon is the spread of what a layer holds, on a log scale
above the meanbelow itthe slice being averaged
One column is one feature, one dot one example · click the left grid to pick a slice
Signal scale at the last layer—
—
View
Normalise each layer
Initialisation
Activation
smaller weightsbigger weights
Values
weight std σ—
σ that keeps the scale—
growth per layer—
signal scale, layer 1—
signal scale, last layer—
gradient scale, layer 1—
layers before it breaks—
a single examplea full batch
every feature alikewildly different
Values
feature spreads, widest ÷ narrowest—
example spreads, widest ÷ narrowest—
mean of everything—
spread of everything—
numbers averaged per slice—
What to observe
- On He with ReLU the ribbon runs flat: the signal reaches the last layer the size it left. On 0.01 or N(0, 1) every layer shrinks or grows it, and 24 layers compound that until it vanishes or overflows.
- Xavier with ReLU shrinks the signal by about 0.71 per layer, because ReLU zeroes half its inputs. He is Xavier with that factor of two put back, a √2 in σ.
- Keep N(0, 1) and turn on Batch norm. Each layer is reset to spread 1, so the weight scale stops compounding: the signal runs flat and the gradient falls from 10¹⁸ to below 100. Layer norm holds it nearer 1.
- Inside one layer, batch norm averages down a column, layer norm across a row. Drag the batch to 1: batch norm outputs only β, layer norm does not notice. That is why transformers use layer norm.
Shortcuts:f fullscreen