Deep Learning
Contrastive Learning
Nobody says where anything goes. The only instruction is that things of a kind belong together and everything else belongs apart, and the embedding arranges itself from that. Four different losses say it in four different ways, and you can watch each one push.
pull it closerpush it awaythe margin
Run to learn · hover any point to look at the loss from there
Nearest neighbour agrees—
—
Loss
sharpsoft
Numbers
loss—
distance within a class—
distance between classes—
constraints still broken—
nearest neighbour agrees—
What to observe
- Pairs pulls every match towards zero and pushes every other pair past the margin. Red arrows vanish as pairs cross the margin; teal ones never do, since a match has no threshold.
- Triplet asks only that a match be nearer than a non-match by the margin. Once that ordering holds the point gets no push at all, wherever the clusters sit.
- InfoNCE uses a softmax instead of a margin. At a lowtemperature nearly all the push comes from the nearest wrong point: hard negative mining for free. Hover a point to read its ruler.
- A class shrinks to a packed cluster, not a dot, only because this page keeps any two points at least a small distance apart: a simplification. In self-supervised learning collapse means every input, of every class, mapping to the same point, which the push between non-matches prevents.
Shortcuts:space run/pause · s step · r reset · f fullscreen