Parameters and weights

A model is a very long list of decimals, and the list is all there is.

Part of the What's inside track on lAItest.

Delete the numbers and you are left with an empty program.

Everything a model knows is stored as decimals in a grid.

A parameter is one number in one of those grids.

Weights set how strongly one number influences the next. Biases shift the result before the bend. Together they are the parameters. Nothing else is stored — no dictionary, no facts table, no copy of the training text. A model "with 70 billion parameters" is a program whose grids hold 70 billion decimals.

A common misconception

Commonly believed: Somewhere in the file there is a compressed copy of the internet the model looks things up in.

Actually: There is not. The training text is gone. What survives is a set of numbers tuned so that the network's guesses would have matched that text. Facts come back as a side effect of guessing well, which is exactly why a model can be fluently, confidently wrong about one.

Parameter count is a capacity, not a score.

More parameters means more room to hold patterns and more arithmetic per word produced. It does not mean better. A smaller model trained on more and cleaner data routinely beats a larger one, which is why size stopped being the headline number vendors lead with.

In one sentence

The model is its parameters. Copy the numbers and you have copied the model exactly.