Deep Learning
Positional Encoding
Attention looks at a set, not a sequence: shuffle the words and every score comes out the same. Position has to be written into the vectors, and the question is whether the score between two words then depends only on how far apart they are or also on where they sit.
query at position pquery at position p′a fast dimension
Click a row of the heatmap to move p
Largest gap between the two curves—
—
Encoding
Query and key
every dimension fastthe last ones very slow
Values
score at offset 0, from p / p′—
score at offset 1, from p / p′—
score at offset 10, from p / p′—
slowest wavelength—
What to observe
- Sinusoidal adds the waves to the word vectors. The score then mixes word with position, so the same offset scores differently from p and from p′: the two curves part.
- Rotary rotates each pair of dimensions instead, and rotations compose: (Rpx)·(Rqy) = x·Rq−py. The two curves lie exactly on top of each other, for any words.
- Drag the base down and every dimension turns fast: the score collapses to a narrow spike at offset 0. A large base keeps slow dimensions that tell distant positions apart.
- A table of one vector per position starts out as noise, shown here untrained. Training shapes it, but it can only cover the positions it was given rows for.
Shortcuts:f fullscreen