silvaan
Simulations/Generative AI/Autoregressive Models
Generative AI

Autoregressive Models

One loop, repeated forever: give every word in the vocabulary a probability, cut the unlikely tail, pick one, append it. The vocabulary here is 40 words instead of 100,000, so you can watch temperature, top-k and top-p act on all of it at once.

Vocabulary—
Seen after this context—
Kept by the cut—
Mass kept—
Last pick—
Surprise of that pick—

Every word of the vocabulary gets a probability at every step, never zero, exactly as in a real model. Top-k and top-p decide how much of that tail is allowed to win.

What to observe

  1. Look at the vocabulary map at the bottom: every one of the 40 words has a bar at every step, even the absurd ones. Words the model has seen after this context get most of the probability, the rest get a sliver. That sliver is the whole reason top-k and top-p exist.
  2. Set top-k to 1. That is greedy: the tallest bar wins every time, nothing is random, and the text falls into a loop. Slide it back up and the loop breaks.
  3. Leave both cuts off and run. The dart can land anywhere, so sooner or later a word from the sliver tail wins and the sentence derails. Any one of them is unlikely, but there are 35 of them.
  4. Top-k keeps a fixed count. When the model is certain, k = 10 drags in nine words it did not believe in; when it is genuinely torn, the same k = 10 may cut off good ones. The count is fixed, the confidence is not.
  5. Top-p instead keeps the fewest words whose probabilities already add up to p. Watch "kept by the cut" as you step: it shrinks to 1 or 2 where the model is sure and swells where it hesitates. Holding the mass fixed is what makes the cut adapt.
  6. Drag temperature without stepping. The pale bars, the model's own probabilities, never move; the solid bars do. Near 0 everything collapses onto the top word, high up they flatten toward uniform. The same numbers, only reshaped before the cut.
  7. The strip is where sampling becomes concrete: the surviving words laid end to end from 0 to 1, each as wide as its probability, and a random number thrown at it. Wide segments win often, slivers win rarely, and that is all "sampling from a distribution" means.
  8. Raise context length. With one word the text wanders; with three it can only replay phrases it has seen. Fluent and unoriginal, or original and unhinged, is the trade every language model lives on.

Shortcuts: space run/pause · s step · r reset · f fullscreen