How AI generates a response, step by step

Between a typed question and a finished answer sit about 7 stages, and the model is only 2 of them. The other 5 are the product and the network around it: building the block of text the model will actually see, carrying it to the provider, fetching documents, running tools, streaming the words back. Most of what separates a good AI product from a poor one happens there rather than in the model.

Walk it stage by stage

Example

Let’s say a support lead asks a company’s internal assistant: what’s our refund window for enterprise plans? 7 words. 2 seconds later the answer comes back, with the clause quoted. What the model received was not those 7 words. It was about 3 pages of text, assembled in that moment, with the question as the last line.

One request · seven stages · step through it

Stage 1 / 7You

You hit enter

Until this moment your message is just text in a box. Pressing enter hands it to the product.

What the request holds right now

Your message

This stage is you.

BackNext stage

The 7 stages

Stage 1, you hit enter. Until this moment the message is just text in a box. Pressing enter hands it to the product: ChatGPT, Claude.ai, an internal tool a team built, whatever surface is in front of you.

Stage 2, the product assembles the prompt. The model does not receive your message. It receives a bundle the product builds: a hidden system prompt (standing instructions the product author wrote: tone, rules, role), the conversation history so far, any documents you uploaded or that the product fetched on your behalf, and last, your new message. All of it stacked into one block of text.

Most of the other stages barely differ between a good tool and a poor one. This one differs enormously: which documents the search returned, how much history stayed in, what the standing instructions actually say. The prompt is not a share of the system. It is the content of this stage, and this stage is where products separate.

Example

Staying with the refund question, here is the bundle in order. The system prompt: a few hundred words the tool’s author wrote once, saying answer only from company policy documents, quote the clause, say so when the policy does not cover the case, never guess a number. The conversation so far: 2 earlier turns about a different customer, still in the bundle because nothing had dropped them yet. The fetched documents: 2 sections of the refund policy and 1 sales FAQ, found by a search over the company’s document store, roughly 900 words, none of them chosen by hand. Then the question, 7 words, at the bottom. Around 1,400 words go to the model, of which 7 were typed.

Stage 3, the bundle travels. That block leaves the device and crosses the internet to the model provider’s servers. Where exactly, and what is retained, is its own subject: where your data goes covers it.

Stage 4, the model reads it as tokens. On the server the bundle is split into tokens, small chunks of text running roughly three quarters of a word each. The whole bundle has to fit inside the context window. If it does not, something was dropped or compressed back at stage 2.

Stage 5, the prediction loop runs. Now the model does its one job: given everything in front of it, predict the next token, append it, predict again. This is the same loop that sits under every model answer, and stage 5 is the only place in the walk where the model is doing what people mean by thinking.

Inside stage 5, on a reasoning model, the loop runs twice over. First it generates its working: a long stretch of text where it sets out steps, tries them and corrects itself. Then it generates the answer that appears on screen. Same loop, same next-token mechanism, pointed first at its own reasoning. The pause you watch, the spinner or the “thinking” line or the summary scrolling past, is that first stretch being produced. Some products show a tidied summary of it rather than the working itself.

It adds no stage. It makes stage 5 longer, and the working is generated text, so it costs time and money like everything else the model produces. A question that would have taken 2 seconds can take 30. That is worth paying where a wrong middle step would ruin everything after it, a calculation with dependencies or a plan built in sequence, and wasted on a question that never needed working. Most products now expose it as a setting on the same model rather than a separate one. How AI models are trained covers how a model is taught to work this way.

Stage 6, tokens stream back. Each token, as the model commits to it, is sent back across the network and rendered on your screen. This is why an answer appears word by word rather than all at once. The streaming is not a stylistic flourish, it is stage 5 made visible. The model’s part ended when the loop finished; the trip back and the rendering are the network and the product again. If the product needs the model to use a tool, a calculator, a search, a database lookup, stage 6 pauses: the product runs the tool, feeds the result back in, and the loop resumes.

Stage 7, the turn is saved. The exchange, your message and the model’s reply, is added to the conversation history the product holds. Which means the next time you hit enter, stage 2 includes this turn. The model itself kept nothing. The thread continues because the product re-assembles and re-sends it, every single turn.

Where the model actually is

The model is stages 4 and 5. Assembling the bundle, moving it, rendering the answer and holding the conversation are the product and the network.

That reframes a lot of everyday questions:

• “Which AI is best?” Partly a question about the model at stages 4 and 5, partly about the product around it at stages 1, 2, 6 and 7. Two products on the same model can feel very different, because their stage 2 and stage 6 differ.
• “The AI forgot what we discussed.” The model did not forget. It has no memory between turns. Stage 2 either did not include the earlier part, or it fell out of the window at stage 4. Memory is a product behaviour, assembled fresh each turn.
• “Why is it slow?” The stages are not equal. Assembly is effectively instant and each network hop is small. The prediction loop dominates, and it grows with the length of the answer, which is why a long reply takes longer than a short one on the same model. A fetched document or a tool call adds a visible pause before anything appears at all. A reasoning model’s working can run longer than the answer it produces.

Why this matters for the business

The model is necessary. Most of what makes a tool good or frustrating in daily use lives in the 5 stages around it.

Stage 2 decides whether the right documents and the relevant history reach the model and the noise stays out. Stage 6 decides whether the tool can look things up and act, or only talk. Stage 7 decides whether a long working session holds together or drifts. Two products built on the same model, priced within a few percent of each other, differ mostly here. The model name is the easiest part of the comparison to read and the smallest part of what is being compared. What gets built around the model is what fills those other 5 stages, and what to ask a vendor about instead of the model name.