Deep Learning

Gradient Descent

The workhorse of training neural nets. Set a learning rate and a starting point, then watch the parameter roll downhill: converge, oscillate, or diverge.

Loss surface

Learning rate 0.10

Start x 2.40

Iteration0

x—

loss f(x)—

gradient—

Loss over iterations

What to observe

On the Convex bowl, raise the learning rate slowly. Aroundη = 1 the steps start to overshoot and bounce across the valley; past η = 2 the loss curve turns upward and it diverges.
The loss plot is the "training curve" you will meet everywhere in deep learning: down and flattening means converging, up means the step is too big.
Switch to Double well and drag the start point across the hump. Thestarting point alone decides which minimum you reach: convexity guarantees one basin, non-convexity does not.

Shortcuts: space run/pause · s step · r reset · f fullscreen