Machine Learning

Overfitting

Fit a polynomial to noisy data and raise its degree. A flexible enough curve can pass through every training point exactly, yet do worse on new data. That gap between memorizing and learning is the central tension in machine learning.

Polynomial degree 3

Regularization λ 0.000

Train error—

Test error—

Error vs. degree (teal train · amber test)

What to observe

Slide the degree up. Train error keeps falling, all the way to zero when the curve threads every point. But watch test error: it bottoms out, then climbs as the curve starts chasing noise.
That U-shape in the mini-plot is the bias–variance tradeoff. Too simple underfits (high bias); too flexible overfits (high variance). The sweet spot is in the middle.
Take a wiggly high-degree fit, then raise λ. Regularization penalizes large weights and smooths the curve back down, recovering generalization without lowering the degree.

Shortcuts: space run/pause · s step · r reset · f fullscreen