Attention as a Soft Lookup

What to observe

1. Swing the query into the fruit cluster. Two or three keys score well, and the output colour becomes a blend of exactly those values. Nothing else is retrieved, but nothing is ever fully excluded either.
2. Drag temperature to the left. The weights sharpen until one key has almost everything and the output is nearly that value alone: a hard lookup. Drag it right and every weight tends to 1/n, the output goes grey, and attention has stopped telling you anything.
3. That is what the √d in the real formula is for. Dot products grow with the number of dimensions, and without dividing by √d the softmax would always land in that first, frozen regime.
4. Watch keys it really uses: it is 1/Σw², the number of keys the output effectively draws on. Near one you have a lookup, near six you have an average.
5. Mask a key by clicking its row. Its weight goes to zero and the others grow to fill the gap, because softmax always renormalises what is left. That single trick is how padding and causal masks work.
6. Notice the query never has to equal a key. It only has to point the right way, which is why attention can retrieve things it has never seen an exact match for.
7. Swap the score for distance. Softmax does not care what it is fed, so attention still works, but what it retrieves changes: −‖q−k‖²/2 is q·k minus ‖k‖²/2, which is the dot product with a penalty on long keys. That is the same fix as normalising, arrived at from the other side. Pick cos θ and the length of the query stops mattering at all, which is why the rings around the query tip collapse back into rays.
8. Nothing on this page says where the query and the keys came from: they are given. Make every one of them a projection of the same sentence and this exact machine becomesSelf-Attention, where the dictionary being looked up is the sentence itself.