Emergent abilities

Some skills appear to switch on suddenly at scale. Whether that is real or an artefact of the metric is still argued.

Part of the How models are made track on lAItest.

A skill the model could not do at all, and then suddenly can. Real phase change, or a trick of the ruler?

The claim.

Wei and colleagues, arXiv 2206.07682, catalogued abilities that are absent in small models, stay absent as you scale, and then appear sharply past some size rather than gradually. Multi-step arithmetic and certain kinds of reasoning were the headline examples. If that is what is happening, it is uncomfortable for planning: you cannot extrapolate from small runs what a larger model will suddenly be able to do.

A common misconception

Commonly believed: Emergence is a settled property of large models: past a threshold, new abilities switch on.

Actually: Schaeffer and colleagues, arXiv 2304.15004, argue much of it is an artefact of the metric. Score multi-step arithmetic by exact match and a model that gets every digit but one scores zero, right up until it gets them all and scores one. Swap to a metric that awards partial credit and the same runs often show a smooth curve. Both papers look at the same models. They disagree about what the measurement means.

What is not in dispute.

The underlying loss falls smoothly with scale, and neither side contests that. The argument is about what happens between that smooth curve and the score on a downstream task. A discontinuous metric can turn gradual improvement into an apparent jump. Whether every reported emergence dissolves under a kinder metric is exactly the open question.

What is the core of the argument against emergent abilities?

Answer: Discontinuous metrics such as exact match can make smooth improvement look like a jump. The rebuttal is about measurement rather than about the models. Exact-match scoring gives no credit for being almost right, so steady underlying progress registers as a flat line followed by a step. This is a live disagreement, so treat any confident claim that abilities do or do not emerge as a position, not a fact.

In one sentence

Everyone agrees the loss curve is smooth. The fight is over whether the sudden-looking jumps in benchmark scores are in the model or in the ruler.