Deep Learning

How Layers Bend Space

A classifier can only draw a straight line. Everything else a network does is bending the space until a straight line is enough. Every hidden layer here is 2 or 3 units wide, so each space it produces can still be drawn.

class 0class 1a linear stepan activationthe classifier's line

Click a thumbnail to see the space that step leaves · with 3 units, drag the big panel to turn the volume

Showing—

—

Through this step: Linear 1 0%

Whole network input

Problem

Units per layer

Hidden layers 3

Activation of each layer

layer 1

layer 2

layer 3

layer 4

Learning rate 0.030

steps—

loss—

accuracy—

weights—

What to observe

Train XOR and press Sweep through. Every Linear step keeps the grid straight, every Activation bends it, until the classifier's single straight line can split the classes (if it stalls, try New weights).
Set every activation to linear. A chain of matrices is one matrix, the grid never bends, and XOR stalls well short: no straight line gets more than three of its four quadrants right.
Rings on tanh with 2 units usually stalls short of 100%, anywhere from 85% to 99%. Each tanh layer is reversible, so the winning region is a bent half-plane that runs off to infinity: at best it squeezes out between the dots, it never truly encloses the inner ring.
Switch to 3 units, even with one hidden layer, and it reaches 100% every time. Drag the big panel: one ring has been lifted out of the plane and a flat plane slides between them.

Shortcuts:space run/pause · s step · r reset · f fullscreen