silvaan
Simulations/Deep Learning/Self-Attention
Deep Learning

Self-Attention

The softmax at the centre of attention is taken apart on its own page. The question here is narrower and stranger: where do the queries, the keys and the values come from? All three are projections of the same X, so the sentence is querying itself. Switch to the vector view for the actual columns of numbers and every multiplication that turns them into an output.

Q, K and V3 views of X
d_k, and its scale—
Looks hardest at—
Keeps of itself—

Click a token or a row of the matrix to move the question. In the vector view, click a column to expand its dot product term by term.

What to observe

  1. The word self is the whole point: Q, K and V are three different projections of the same X. Nothing external is being looked up, the sentence is querying itself.
  2. Switch to the vectors. Every column is one token: its embedding x on top, the key Wkx under it, then the dot product with the single query column on the left, then the softmax weight, then the value Wvx it hands over. The output is those value columns added up in exactly those proportions.
  3. Pick the head called one before and look at its rows in the vector view: the meaning dimensions are all zero. That head has chosen to read position and nothing else, and it does it with the same dot product as every other head.
  4. Now pick verb → noun and click chased. It reaches back past the determiner to cat. The output column turns noun-coloured, which is what it means to say a verb has pulled its subject into its own vector.
  5. Turn on the causal mask. The upper triangle goes dark, every row renormalises over what is left, and the first token is stuck attending only to itself. That is the only difference between an encoder and a decoder here.
  6. Nothing here tunes the softmax: the only scale is the √dkthe formula already has. What sharpening, flattening and masking do to those weights is the subject ofAttention as a Soft Lookup, where the keys are a fixed dictionary rather than the sentence itself.

Shortcuts: space run/pause · s step · r reset · f fullscreen