silvaan
Simulations/Deep Learning/Softmax & Temperature
Deep Learning

Softmax & Temperature

A network ends in a row of raw scores that mean nothing on their own. Softmax turns them into probabilities, and one number decides how strong an opinion it forms: temperature takes the same scores from a single confident answer to a shrug.

the winning classthe others

Drag a score up or down

The winner takes—
—

picks onegives up
Scores
winner—
its probability—
classes it really uses—
entropy—

What to observe

  1. Drag add the same to every score. Every score slides and not one probability moves: softmax reads only the differences.
  2. Take temperature down to 0.1. The gaps are divided by a small number, so they grow and the winner swallows everything: softmax becomespick the largest.
  3. Take it up instead. The gaps are divided away, every class heads for 1/5, and classes it really uses (1 over the sum of squared probabilities) climbs towards 5.
  4. On Two are close, the two leaders trade almost all the mass as τ moves while the rest stay small: temperature spreads attention only among what was already competitive.

Shortcuts:f fullscreen