Statistics

Information and Entropy

A bit is the answer to one yes/no question. Finding one card in 16 takes four of them; news that was unlikely takes more questions to pin down than news you expected. Entropy is the average number of questions: how much a source really tells you, and how far its messages can be compressed.

width: how likelyheight: questions to pin it down, log₂ 1/plevel: the average, entropy H

In Explore, drag the bars on the right to change the odds

Now—

—

Chapter

Outcomes

Exact values

entropy H—

best yes/no code (Huffman)—

most possible, log₂ N—

What to observe

Each yes/no answer halves what is left, so one card in 16 takes 4 questions: 16 = 2⁴. One in a million takes only 20.
When the odds are uneven, ask about the likely outcome first. Sun (½) is settled in one question, rain (⅛) needs three: an outcome with chance p needs log₂ 1/p questions.
Send the weather every day this way and it costs 1.75 bits a day on average, not 2. Pour the steps flat and the level is the entropy.
A fair coin carries 1 bit a flip. A coin that lands heads 99% of the time carries 0.08: you could guess it anyway.
Entropy is largest when nothing is favoured: with N equal outcomes it is log₂ N, the same count of possibilities as Boltzmann's S = k ln Ω.

Shortcuts:space run/pause · r reset · f fullscreen