Base models and instruct models

The model that finishes pretraining is not the model you talk to. It is the raw material.

Part of the How models are made track on lAItest.

Ask a base model "What is the capital of France?" and a perfectly good answer is "What is the capital of Germany?"

It is not broken. It is doing its job exactly.

A base model continues text. An instruct model answers you.

Base weights come straight out of pretraining. They model how documents flow, so a lone question is most naturally followed by more questions from the same quiz. An instruct model is that same base, trained further on examples of requests paired with good responses, until answering becomes the most likely continuation. Same architecture, same starting weights, different behaviour.

A common misconception

Commonly believed: Instruct models are smarter than base models. The extra training makes them know more.

Actually: The extra training adds almost no knowledge. Nearly everything the model knows was already there when pretraining ended. What later training changes is which of the many possible continuations comes out: an answer instead of an echo, a refusal instead of a recipe. It shapes behaviour, not the store of facts.

Whether you can get the base model is a licensing question, not a technical one.

Most hosted models are only ever exposed as instruct-tuned endpoints; the base weights stay inside the lab. When a lab publishes open weights it sometimes publishes both, and researchers use the base as a clean starting point for their own tuning. Nothing about the architecture decides this. The release policy does.

Why does a base model often answer a question with another question?

Answer: It predicts the most likely continuation, and a list of questions continues with questions. Base models optimise one thing: what token plausibly comes next in a document. A lone question looks like the opening of a quiz, so more quiz is a good prediction. Answering only becomes the likely continuation after the model has seen many examples of requests being answered.

In one sentence

A base model is a text continuer. Everything that makes a model feel like an assistant was added after pretraining finished.