Why it makes things up

Confident invention is not a glitch bolted onto the machine. It is the same mechanism that makes it work.

Part of the What is this thing? track on lAItest.

A model will produce a book, an author, a page number and a quotation, and none of the four will exist.

It is not lying. It has no idea it did that.

Plausible is the only target

The machine produces likely continuations. A real citation and a fake one built to the same pattern are nearly identical as text: same shape, same rhythm, same kind of words. There is no separate step that checks whether the thing produced is also true, because there is nothing inside to check it against. Truth was never the objective. Plausibility was.

A common misconception

Commonly believed: Hallucination is a bug, and a good enough model will stop doing it.

Actually: It is structural. The same ability that lets a model finish a sentence about something it has never seen is the ability that lets it invent a citation. You can cut it down a great deal — hand it the source text, allow it to say it does not know, make it check its own work, keep it away from questions it cannot ground — but a system whose job is producing likely text will sometimes produce likely text that is false. Manage it; do not wait for a patch.

Where the risk climbs

It climbs wherever the true answer is rare in text: exact figures, dates, legal citations, small local details, obscure people, anything after the cutoff. It also climbs when your question assumes something false, because the most likely continuation of a false premise is text that goes along with it.

Which request is most likely to come back invented?

Answer: List the page numbers where a contract mentions payment terms, without the contract. The other three have their answer sitting right there in the prompt. The second asks for exact facts about a document the model cannot see, so the only thing it can do is produce something shaped like page numbers.

In one sentence

It cannot tell remembering from inventing, because it does neither — it produces likely text, and likely text is sometimes false.