Why AI hallucinates (makes things up)

A model has no way to tell a true sentence from a plausible one. Both come out of the same next-word prediction, in the same voice, at the same level of confidence. So hallucination is not a separate failure bolted onto an otherwise reliable machine. It is the machine working normally on a question where plausible and true come apart.

The mechanism

A model produces the next word given everything before it. There is no separate fact-checker in the loop. The question it answers is “what word probably comes next?”, not “is the sentence I am producing true?”

Where the training data held thousands of paragraphs of one shape, the model completes that shape even when it was never shown the particular fact being asked for. It writes the most plausible filling, fluently, in the same voice as every accurate paragraph it has ever produced.

A true answer and a fabricated one come out of the same process. They look identical from outside because on the inside they are the same kind of thing.

Example

Let’s say someone asks a model for the details of a recent FDA approval. Back comes a drug name, an approval date and a drug class, written as steadily as the 20 answers before it. The approval never happened. Nothing in the sentence reads differently from a sentence about an approval that did.

Where the failure concentrates

Hallucination is not spread evenly across questions.

Specific numbers and dates. The model is reliable on the shape of a sentence about a number. The digit itself is far less reliable, especially for figures that were not repeated widely across the training data.

Names and citations. “According to a recent paper by Singh and Patel” reads as authoritative, and the paper may not exist. The form of a citation is regular and easy to produce. Who wrote what, and where, and when, are individual facts the model often does not hold.

Anything recent. Every model has a knowledge cutoff, the date its training data ends. Past that date it is guessing from the patterns of what came before, and what models actually know goes deeper into that gap. The chat products now search the web for questions like this, which moves the failure rather than removing it. The answer is only as good as the page that came back, and a model handed nothing useful still writes the sentence.

Anything internal to a single company. A company’s own data was not in the public training set. The model has nothing specific to draw on, so it matches against how comparable companies have been described in public, and the specifics come out wrong.

Long, confident, detailed answers. In human writing, length and specificity are signs of effort and expertise. In a model’s output they are neither. A 2-paragraph answer with 7 specific details is no harder to produce than a one-line answer, and each extra detail is another chance to be wrong. Models do hedge, and the hedging is trained: it tracks topics the lab wanted handled carefully, medicine and law and money, not whether the name in this sentence was invented 30 words ago.

What reasoning models change

Ask a current model something involved and it works through the problem before answering. That is a trained behaviour, covered in how AI models are trained, and it does catch some of this. A model that checks its own steps can notice that the figure it produced in step 2 does not match the one it used in step 4. On problems with several steps the current generation is meaningfully stronger than the one before it.

What the working cannot do is supply a source. Checking an argument against itself finds contradictions inside the argument. It cannot tell the model that the drug approval it just described never happened. Where the answer turns on a fact the model never held, more working buys a better-argued wrong answer.

Why grounding changes the picture

Put the actual approval letter in front of the model and ask the same question, and the mechanism shifts. The model is no longer reaching for patterns from training. It is summarising text sitting in its context window. The next-word prediction now runs against the document, and the most likely next words become the words that match it.

Putting the answer into the model’s view before asking the question is what every serious factual-AI product is built on. Pasting a document into ChatGPT, uploading a PDF, connecting a knowledge base, enterprise “AI search”: all variants of the same move. The industry words for it are retrieval and grounding, and tools and memory walks through how it is wired up in real systems.

Grounding does not make hallucination impossible. A model can still misread a source, mis-summarise it, or smooth over a contradiction inside it. The failure changes from inventing a fact out of nothing to misrepresenting something that exists, which is smaller and much easier to check. It also moves the weak point. When a grounded system answers wrongly, the usual cause is the wrong passage coming back, not the model, and a wrong passage produces an answer that looks exactly like a right one.

Example

Now say a support assistant is wired to the company’s policy library, and a customer asks about returning a clearance item. The search brings back the general returns policy instead of the clearance one. The model answers accurately from the page it was handed, and the customer is told they have 30 days when they have none. The model read its source correctly. What failed was which page came back.

The shape of the risk

Hallucination tracks two things: how much the model actually holds on the question, and how much checking happens before the output is used.

Some topics sit in the training data from a thousand angles. Common concepts, well-documented technical material, generic business writing. There the patterns are mostly correct and ordinary review catches the rest. Drafting an internal email about scheduling a meeting is that end.

At the other end the model holds nothing specific. The company’s own numbers, an event from last week, a person who is not a public figure, an obscure regulation. There it fabricates steadily unless the relevant text is put in front of it first. Quoting a customer back to themselves, citing a clause from a contract, stating last quarter’s revenue: that end.

The same model handles both. Newer models fail less often, and a reasoning model fails less often again where the middle of the answer can be checked. The shape does not change. What an answer is worth depends on what is being asked and what is in front of the model when it is asked, far more than on which model is answering.

The expensive mistakes are rarely the obvious ones. A fabricated answer that reads like a briefing note gets reviewed the same way a correct one does, and passes. Output right in form and wrong in a part nobody stops on is what “almost true” names, and it is where the cost lands.