Statistics
KL Divergence
Every outcome surprises you by −log p(x): little when it was likely, a lot when it was rare. Believe the wrong distribution Q while the world draws from P and you are surprised a bit more on average. That average extra surprise is KL(P‖Q): never negative, zero only when Q is right, and not the same the other way round.
P: how the world really drawsQ: what you believeextra surprise (Q too low)less surprise (Q too high)
In chapter 3, drag Q sideways to move it and up or down to change its width
Now—
—
Chapter
Exact values
entropy H(P)—
cross-entropy H(P, Q)—
KL(P ‖ Q)—
KL(Q ‖ P)—
What to observe
- Outcomes near the peak of P barely surprise; the rare ones out in the tails surprise a lot. The average settles on the entropy.
- With the wrong belief Q, some outcomes are less surprising than they should be (blue) and more are more surprising (red). The average extra never ends up below zero.
- Drag Q onto P and all the red and blue vanish: KL is zero only when Q = P. Make Q narrow and watch the red in the tails explode.
- Q sits on one hump of P. Draws from P on the other hump shock it, so KL(P‖Q) is huge; draws from Q are never a shock to P, so KL(Q‖P) is just one bit.
- Minimising KL(P‖Q) forces Q to cover everywhere P goes, so it spreads over both humps. Minimising KL(Q‖P) only punishes Q for going where P does not, so it settles on one hump.
Shortcuts:space run/pause · r reset · f fullscreen