Deep Learning

Batch Size & Gradient Noise

Nobody computes the real gradient. A handful of examples is drawn at random, each one pulls in its own direction, and their average is the step. Watch which examples were drawn, watch the arrow they add up to, and watch how much it misses the true gradient by.

the gradient over all 256 examplesthe examples you drew, and each one on its ownthe average of them, which is the stepwhere the run has been

Hover an example to see the direction it pulls · click the surface to start a run from there

This batch against the truth—

—

Batch size 8

one examplethe whole set

Learning rate 0.30

Each drawn example pulling on its ownThe other batches you could have drawn

Values

examples drawn—

this batch points off by—

error of this draw—

typical error of a draw—

size of the true gradient—

noise ÷ signal—

radius it settles into—

steps taken—

What to observe

Press New batch. A handful of examples light up, each adds its own violet arrow, and their average (indigo) is the step. The teal arrow is the true gradient over all 256, which nobody computes.
At batch size 1 the step can point almost the wrong way. The dashed ring is the spread of the average and shrinks like 1/√B: sixteen times the work buys only about four times less noise.
At 256 the batch is the whole set. The factor √((N − B)/(N − 1)) for drawing without replacement reaches 0, the ring closes and indigo lies on teal.
Run with a small batch. It never stops at the minimum but jitters in aball around it, whose size is set by the learning rate against the batch size.

Shortcuts:space run/pause · s step · r reset · f fullscreen