What a context window is, and what happens when it fills up

The context window is the most text an AI model can take in at once, and on each turn it is everything the model sees. The model keeps no record of a session: every turn it is handed one block of text, up to that size, and runs the next-word loop over whatever the block contains.

So an instruction given earlier only shapes an answer if it is still inside the block on that turn. Nothing carries over on its own, and when something drops out the model does not notice it is gone.

How the window works on each turn

Every turn, the product assembles one block of text and sends the whole thing. It holds the latest message, the earlier messages in the chat, the model’s own earlier replies, any document uploaded or pasted, anything a search or a tool call pulled in, and a hidden set of instructions from the product (the “system prompt”) that tells it how to behave.

The model reads that block and writes the next reply. Then the block is thrown away and rebuilt from scratch on the following turn. Nothing persists inside the model in between. (How ChatGPT and other AI assistants work walks the whole path from keypress to answer.)

What a token is

A model does not read whole words. It reads short chunks, some of which are whole words and some of which are pieces. “The” is one chunk. “Fundamentals” is about 3. Those chunks are called tokens, and they are the unit everything about the window is quoted in.

The rough conversion for English is 4 tokens for every 3 words. A thousand-word memo is about 1,300 tokens. A 250-page book is around 130,000.

Tokens are also the unit of the bill. The companies that sell access to models charge by the token, on the way in and on the way out, and what that comes to for a real workflow is mostly a question of how much text each run carries.

See it fill up

Context window: 126 / 600 tokens

System

You are a helpful assistant for a customer support team.

24t

User

A customer was double-charged on their last invoice. How do I handle this?

38t

Assistant

Start by confirming the duplicate charge in your billing system, then issue a credit to the customer with a brief explanation…

64t

+ Short message · 32t+ Long message · 124t+ Paste a policy section · 400t+ Retrieved tickets · 260tReset

Add messages. When the window fills up, watch what happens.

The window here is tiny so that the limit arrives in a few messages. A real one holds far more and behaves the same way. Text goes in until there is no room, and then something already in there has to go.

How big the window is now

The current flagship models from the major labs all take roughly a million tokens in one request, which is about 750,000 words, or a couple of thousand pages. Cheaper tiers and older releases hold a fraction of that, commonly a few hundred thousand.

What a product hands the model is a different number. Chat products often give a conversation less than the model’s full window, and the amount can vary by plan. Current assistants keep far more of a conversation in view than earlier ones did, and that space also has to hold the system prompt, uploaded files, search results and tool output. When a conversation runs long, the product trims or summarises older turns to make room. That is why a long session can run out of room before the model would, and why a morning’s careful briefing can drop out of view well short of the model’s own limit.

Reading and writing capacity are not the same. A model that accepts a million tokens going in caps a single answer far lower, typically near a tenth of that. It can read a book and cannot write one back in one go.

• A 50-page report, around 20,000 words. Fits anywhere with room to spare.
• A small codebase, around 200,000 words. Fits in the largest current windows.
• A year of company email, several million words. Does not fit, and a bigger window is not the fix.

The window keeps growing. “Fits” is still not the same as “works well”.

Why a bigger window is not automatically better

Accuracy on the relevant part falls. Models perform worse on long contexts than on short ones, a decline known as context rot. The detail that matters gets buried in the detail that does not, and what comes back is fluent and slightly off rather than obviously wrong, which is the hardest kind of error to catch. This has improved a great deal and has not gone away.

Answers take longer. A full window is more text to get through before the first word comes back. Mildly annoying in a chat, and it compounds in anything that runs in a loop.

Cost rises, though less simply than it used to. Sent cold, a 100,000-token prompt costs roughly 50 times what a 2,000-token one does, and the largest windows carry a price step partway up. But text re-sent unchanged, which is exactly what a fixed document or knowledge base is, is now billed at a fraction of the first-time rate by every major provider. Pushing a big document through every turn is still slower and still less accurate. It is no longer expensive in the way it once was.

A tight, relevant prompt of 2,000 words beats a dumped-everything prompt of 200,000. Almost always, and now for reasons that have little to do with the bill.

What happens when the window fills up

A long session eventually runs past the window. What happens next is a choice made by whoever built the product.

It drops the oldest turns. The model stops seeing the start of the conversation. It is not forgetting; that text simply was not in the block on this turn.

It summarises the older part and keeps the summary. That buys room, and the summary is thinner than what it replaced. What gets compressed away is the specific detail, not the general shape.

It refuses. The request comes back as an error, or a long upload is rejected outright.

From outside, the first two are almost impossible to tell apart, and both feel like an assistant that has quietly stopped following an instruction it was given hours ago. The product rarely says which one it did. You get a slightly worse answer with nothing to tell you the input changed.

Why every new chat starts blank

A new conversation contains none of the previous one. No memory of yesterday, none of the person typing.

The major chat products ship a separate “memory” layer that stores facts between chats: a small file of notes the product writes, then puts into each new conversation’s block. The model still has no memory. The product is doing the remembering, and what it writes down is a design decision rather than a property of the model.

Where this lands in real systems

Which messages stay, which get summarised, which get dropped, when a fresh window starts, what the summary is told to preserve: those decisions shape the experience more than which model is wired up underneath.

Example

Let’s say a support assistant is 40 messages into a customer thread. The window holds the system prompt, the account record, the refund policy and the conversation so far. Around message 30 the conversation alone outgrows the space allotted to it, so the product summarises messages 1 to 20 into a paragraph and keeps the rest word for word.

The paragraph records that the customer reported a double charge and a credit was issued. It drops the part where the customer said a credit was unacceptable and asked for the money back. On message 41 the assistant offers a credit. The customer has now been offered the same wrong resolution twice, and nothing in the transcript looks like a fault.

Nobody chose to lose that sentence. Someone chose what the summary was told to keep, and the sentence was not on the list. Deciding what goes into the window is most of the work in building anything that runs for a long time.

Common questions

What is a token in AI?

A token is the small chunk of text an AI model reads and writes: often a whole short word, sometimes part of a longer one. In English, 100 tokens is roughly 75 words. The limit on how much a model can take in at once is measured in tokens, and AI providers charge by the token for both the text sent in and the text that comes back.

What is context rot?

Context rot is the drop in an AI model's accuracy as the amount of text it is given grows. The detail that matters gets buried among detail that does not, and the answer comes back fluent but slightly off rather than obviously wrong. Newer models handle long inputs much better, but the effect has not gone away, so a short, relevant prompt usually beats pasting in everything.

How big is the context window in ChatGPT and Claude?

The largest current models from the major AI companies accept around a million tokens in one request, roughly 750,000 words. Cheaper and older models take less, often a few hundred thousand tokens. The chat apps usually give a single conversation less than the model's full limit, and the amount varies by plan, so a long chat can run out of room well before the model would.

What happens when the context window is full?

It depends on the product. Most drop the oldest messages, so the model stops seeing the start of the conversation; some replace older messages with a shorter summary; some refuse with an error. The first two rarely announce themselves, so the usual sign is an assistant that quietly stops following an instruction given much earlier.

Key takeaways

• The context window is the text a model is shown on a single turn. It is not memory. Nothing carries over on its own.
• The window is measured in tokens, the small chunks of text a model reads and writes. Roughly 4 tokens for every 3 English words.
• Current flagship models take around a million tokens in one request, which is a couple of thousand pages. Cheaper tiers and older releases hold a fraction of that, and chat products often give a single conversation less than the model's full limit.
• A full window is not a well-used one. Accuracy on the part that matters often falls as more goes in, and the answer takes longer.
• Money is the weakest argument for short prompts now. Text that is re-sent unchanged is billed at a fraction of the first-time rate.
• When a session runs past the window, the product drops the oldest turns, summarises them, or refuses. From outside the first two are indistinguishable, and neither is announced.
• Any memory across chats is the product writing notes and putting them back into the next conversation. The model has no persistence of its own.
• Deciding what goes into the window is most of the engineering in a long-running AI product, and it matters more than which model is underneath.