Optimizers
Five optimizers descending the same loss, drawn two ways. As a surface, the loss is a height you can read off directly and click on to choose where they start. As a volumetric cloud, the loss is minus a density and the balls move freely through 3D space toward the brightest cores. The hyperparameters carry across, so the same setup can be watched in both.
Drag to orbit, scroll to zoom. On the surface, click to drop the start point there; in the cloud, colour runs blue (sparse) to red (dense) and the loss is minus the density, so the balls climb into the bright cores. Toggle an optimizer with its coloured checkbox. The dashed white arrow is the downhill gradient; the coloured arrow is its momentum.
What to observe
- Pick the Ravine surface. Plain SGD zig-zags across the steep walls while Momentum and Adam power straight along the valley floor.
- On Two wells, click to start on the ridge between them. A small nudge decides which minimum each ball falls into: that is non-convexity.
- Turn on both arrows. The white dashed gradient always points straight downhill, but the coloured momentum arrow lags behind: that stored inertia is why momentum overshoots and then corrects.
- Now switch to the density cloud without touching anything else. The optimizers are the same five, running the same hyperparameters, but the loss now lives in three parameters instead of two and there is no height to read off: only the haze tells you where the minima are. Watch the same learning rate that behaved on the surface start to overshoot.
- In the cloud, orbit while a path is running. A trail that looks like it is diving into a bright core often turns out to be passing well in front of it, which is the honest version of what "seeing" a loss landscape means once you are past two parameters.
- Hit new field a few times. With several cores, different optimizers can end up in different ones from the very same start.
Shortcuts: space run/pause · s step · r reset · f fullscreen