Deep Learning
Neuron Activations
A network is trained here, in front of you, on shapes drawn with the same brush you are about to use. Then draw one yourself and watch it go through: every pixel you light up sends its value down a hundred wires, each wire multiplies it by its own weight, and every neuron adds up what reaches it. Click any neuron to see the picture it has learned to look for.
only the loudestall of them
Epoch—
Loss—
Accuracy, unseen—
It says—
Drag on the grid to draw. Click any neuron to open it up: a hidden unit of the first layer shows the whole picture it is looking for, deeper ones show which units below they listen to.
What to observe
- Draw a single vertical stroke, then drag it slowly sideways. The output barely flinches. Now tip it over towards horizontal ten degrees at a time. The answer does not slide, it hands over: vertical holds, then rising takes it, then horizontal, because those are the only answers it has anywhere near that angle. Somewhere on the way it may jump, briefly and with total confidence, to something absurd.
- Click a neuron in the first hidden layer. Its incoming weights are one number per pixel, so they are a picture, and it is drawn below the grid. Several of them come out as smudged strokes at some angle: nobody asked for edge detectors, they are just what minimising the loss produced.
- Turn connections drawn down to a few percent. What is left is the handful of wires actually carrying the decision. Count the units markedoff: for any one stroke about half the first layer sits at exactly zero, and a ReLU that is off sends exactly zero down every wire leaving it, however large the weight on that wire happens to be.
- Watch the wires while it trains, and watch the epoch counter. Almost all of the movement is over in the first two or three epochs: the first layer alone changes by more than half its own size in epoch one, and everything after that is polishing. The loss keeps falling long after the picture has stopped visibly changing.
- Erase until only a few pixels are left. The answer does not fade towards "nothing", it stays above 90% on whichever of the six is nearest. A softmax has to divide one unit of probability among its outputs, so a classifier trained on six shapes has no way at all to saynone of these, and confidence is not evidence that it was asked a fair question.
- Press train again a few times. The accuracy lands in the same place, but the pictures inside are different every run. What a particular neuron means is an accident of where the weights started, and only the layer as a whole has a fixed job. The geometry behind that job is the subject ofHow Layers Bend Space.
Shortcuts: space run/pause · s step · r reset · f fullscreen