Optimizers

What to observe

1. On the 1D slice ½x², plain GD multiplies x by 1 − η each step. Atη = 1 it lands on the minimum in one step, above 1 it overshoots and bounces, above 2 it diverges and the loss strip climbs.η = 1 η = 1.5 η = 2.1
2. On the Ravine, GD drops onto the valley floor and then crawls along it. Momentum and Adam run along the floor and arrive long before.Show me
3. On Ripples, AdaGrad and RMSProp take the same step, gradient over a root of squared gradients. AdaGrad's sum only grows, so it lags far behind; RMSProp forgets with ρ and keeps its pace. Show me
4. In the density cloud the loss lives in three parameters and only the haze shows the minima. Orbit while it runs, and hit New field: from one start, different optimizers can end in different cores. Show me