How Layers Bend Space

What to observe

1. Start on XOR and let it train. The top row is the whole pipeline, one panel per step: the input, then a Linear and anActivation for each layer, then the Classifier. The only thing the classifier ever draws is a straight line; every panel before it is the work done to make that line sufficient.
2. Click back to input space. The same boundary is now a curve, because you are seeing it through all the bending in reverse. A network does not learn curved boundaries, it learns a change of coordinates in which the boundary is straight.
3. Stop on any panel marked Linear. The grid there is still a perfect grid: rotated, stretched, sheared, slid sideways, but every line still straight and every pair of parallels still parallel. That is the entire contribution of the matrix, and it is why stacking matrices alone gets you nowhere.
4. Now step forward into the Activation that follows it, and watch the straight lines finally bend. Press Sweep through to run the whole network as one continuous motion and the rhythm is unmistakable: rigid, bend, rigid, bend, until one line is enough.
5. Pick a step and drag how far through this one step slowly. Halfway through a Linear step the space is exactly halfway there, because the average of two matrices is another matrix. Halfway through anActivation it is not, and that is the difference in one sentence.
6. Try Blobs. The classes are already separable, so the layers barely move and the grid stays almost square. The amount of bending a network does is not a setting, it is whatever the problem demands.
7. Try Rings with everything on tanh. It climbs to about 90% and then stops, and leaving it running all afternoon will not help. Every tanh layer is reversible: it bends the plane but never glues two places together. The region the classifier claims is therefore always the bent image of a half-plane, which runs off to infinity and can never be a bounded blob. It cannot surround the inner ring without also cutting through the outer one.
8. Switch those layers to leaky relu and give it a few thousand steps. It usually does reach 100%, sometimes only after a restart. The grid shows why: a relu-family activation puts a hard crease in the plane and squashes one side of it by tenfold, and a ring flattened onto its own crease can be threaded by a straight line, slipping between the sampled points where the smooth warp could not.
9. Now the honest answer to why Rings is hard. Put it back on tanh, where it was stuck around 94%, and switch to 3 units. It closes at 100%, on the first try, with a single hidden layer. Drag the big panel to orbit: the layer lifted one ring out of the plane and a flat plane slides between them. Nothing had to be folded, there was simply somewhere else to go.
10. That is what width buys, and it is not subtle. On Rings, 2 units reach 100% in about 1 run in 12; 3 units reach it in 12 out of 12. On XOR it is 4 in 10 against 10 in 10. Depth bends the space, width decides how much room there is to bend it in.
11. Island and Onion are the same trap in different clothes: a filled island in a sea, and three nested rings. Each needs the winning region to be a closed blob, and a reversible warp of a plane cannot make one. Onion is the sharpest of the three, and worth doing in this order: at 2 units it hovers around 70%, at 3 units it jumps to about 98%.
12. None of this is the network memorising the dots. Trained on 240 points and then asked about 2400 fresh ones from the same shape, the 2-unit runs hold their plateau (Rings 87%, Onion 65%) and the 3-unit runs keep their answer (Rings 100%, Onion 90%). The extra dimension buys a real boundary, not a thread stitched between the samples.
13. Plain relu is worth trying precisely because it is the shakiest of the three here. With only two units, one of them can land where its input is negative for every single point: it outputs a flat zero, its gradient is a flat zero, and it never comes back. Half the space collapses onto a line and the run is over before it started. That is a dead unit, seen from the inside, and it is why leaky relu exists.
14. Set every activation to linear. However many layers you stack, the grid stays a grid: a chain of linear maps is one linear map, so the boundary is stuck being straight and XOR stalls in the sixties, which is all a straight line can do on it. The squashing is not a detail, it is the entire source of a network's power.
15. Try Spiral, the one that stays hard. Two units plateau in the seventies; three units get to about 92% and still rarely close it. A spiral needs to be unwound, not just lifted, and that takes more room than three dimensions. Real networks use hundreds of units per layer for exactly this reason, which is also why they can no longer be drawn.
16. Press New weights a few times on the same problem. Sometimes it finds the fold in a second, sometimes it never does. At this width the starting point genuinely decides the outcome, which is why real training runs are restarted and why width buys reliability, not just capacity.