Neural network

Layers of multiply, add and bend — repeated until the numbers mean something.

Part of the What's inside track on lAItest.

Nothing inside a language model is a neuron.

The name is a 1940s metaphor that stuck.

It is a stack of multiplications.

Text becomes a list of numbers. Each layer multiplies that list against a grid of other numbers, adds a small offset, and bends the result through a simple curve. Do that a hundred times over and the final list is a score for every word that could come next. That is the whole machine.

A common misconception

Commonly believed: A neural network is a simulated brain, so it must think roughly the way we do.

Actually: It borrows exactly one idea from biology: many simple units, each combining signals from the layer before. Real neurons also have timing, chemistry and physical growth. None of that is here. The maths is closer to an enormous spreadsheet than to a mind.

The bend between layers is what makes depth worth anything.

Multiplying and adding, over and over, can only ever describe a straight-line relationship. Between layers, every number passes through a small non-linear function that clips or curves it. That step is the only reason a hundred layers can describe something a single layer cannot.

Why does a neural network put a non-linear step between its layers?

Answer: Without it, the whole stack collapses into one straight-line step. Stacked multiplications are still one big multiplication. The bend is what stops a deep network from being mathematically identical to a shallow one — it is where depth stops being decoration.

In one sentence

A neural network is multiply, add and bend, repeated until the numbers at the end mean something.