Catastrophic forgetting

Train a model hard on your data and it can quietly lose abilities nobody thought to test.

Part of the How models are made track on lAItest.

Fine-tune a model on your house style and it will learn your house style. It can also get worse at arithmetic.

Nothing broke. This is how the training works.

There is one set of weights, and every ability shares it.

The model has no compartment for tone and a separate one for arithmetic. Every ability is spread across the same numbers. Training pushes those numbers toward whatever the current data rewards, and nothing in the process protects an ability the current data never exercises. Push hard enough for long enough on a narrow set of examples and skills outside that set drift away. That drift is catastrophic forgetting.

A common misconception

Commonly believed: Forgetting means my data was bad or my hyperparameters were wrong.

Actually: It shows up with clean data and sensible settings. It is the default behaviour of gradient descent on a narrow distribution, not a bug you can configure away. What you can do is lower the pressure: fewer passes over the data, a smaller learning rate, general examples mixed back in, or freezing the base entirely and training an adapter beside it.

Related, and not the same: overfitting.

Overfitting is the model memorising your training examples and doing worse on new examples of the same task. Forgetting is the model doing worse on a different task entirely, one your eval set probably does not cover. Overfitting shows up in your validation numbers. Forgetting shows up in production, in a capability nobody thought to measure.

Which is why the eval set comes before the training run.

Before any tuning, write down what the model can already do that you need it to keep doing, and measure it. Then measure the same list afterwards. Without that baseline you cannot tell a model that improved from a model that improved at one thing while losing three others.

In one sentence

Teaching a model something new can cost it something old, and you only find out if you measured the old thing first.