Machine Learning

Overfitting

Fit a polynomial to noisy data and raise its degree. A flexible enough curve threads every training point, and does worse on new data.

training pointstest pointsthe fittrue functiontrain error

Test error—

—

Polynomial degree 3

Regularization λ 0.000

train error—

test error—

What to observe

Slide the degree up. Train error keeps falling and reaches zero at degree 11, where the curve threads all 12 training points.
Test error in the strip falls, bottoms out, then climbs as the curve chases noise: the bias–variance tradeoff. The best degree sits in the middle.
At degree 11, raise λ. Penalising the weights (all but the intercept w₀) smooths the curve and brings test error back down without lowering the degree.

Shortcuts:f fullscreen