A context window is all the text a model can reference at once when generating a response: the system prompt, the entire conversation so far, and the response it's currently generating, all counted together in tokens. It isn't the model's long-term knowledge, that's fixed at training time, it's closer to working memory for a single request.
Learning Objectives
- Explain what counts toward a context window, and what happens when a request exceeds it.
- Explain why a bigger context window isn't automatically a better one.
- Distinguish a context window from persistent memory.
What Actually Counts Toward It
System Prompt
Conversation History
every prior turn
Context Window
one shared budget
Generated Response
counts too
The system prompt, every message in the conversation so far, and the response being generated all draw from the same fixed token budget. A long system prompt or a long conversation history leaves less room for everything else, including the model's own answer.
What Happens When a Request Exceeds the Limit
| Situation | What happens |
|---|---|
| The input alone already exceeds the limit | The request is rejected outright with an error before generation even starts |
| The input is fine, but the response is capped so short generation could hit the limit | The model generates until it reaches the limit, then stops early with a specific reason indicating the context window was hit, not a specific completion |
Count Tokens
before sending
Compare to Limit
Trim if Needed
Send Request
Neither case is a silent failure, both are explicit, detectable outcomes, which is exactly why real applications check token counts ahead of time rather than discovering a limit mid-response.
Why Longer Isn't Automatically Better: Context Rot
A larger context window lets a model handle longer conversations and documents, but stuffing more into it isn't free. As token count grows, a model's accuracy and recall measurably degrade, a documented effect sometimes called context rot. Information buried in the middle of a very long context is generally recalled less reliably than information near the start or the end.
This is why curating what's actually in context (trimming irrelevant history, summarizing older turns) is often more valuable than simply having access to a larger window.
Context vs Memory
- Exists only for the duration of one request or one active conversation
- Disappears the moment a new, separate conversation starts
- A fixed, known size the application can measure and manage
- Something an application deliberately builds on top of the model, not a built-in model feature
- Survives across separate conversations, by explicitly storing and re-supplying information later
- Typically implemented with an external store (a database, or a RAG-style retrieval step)
A model has no built-in way to recall an earlier, separate conversation. Anything that looks like "remembering you" across sessions is an application deliberately re-injecting stored information into a new context window, not the model retaining anything on its own.
Common Beginner Mistakes
- Assuming a model remembers previous separate conversations by default, without an application explicitly re-supplying that information.
- Sending the entire conversation history on every turn without trimming it, and being surprised the request eventually gets rejected.
- Assuming a bigger context window always produces better answers, when accuracy can actually degrade as irrelevant content piles up.
- Forgetting the response being generated also draws from the same token budget as everything sent in.
FAQ
If I ask a model to summarize a huge document, does the whole document need to fit in the context window?
Yes, if it's sent directly in the prompt. For documents too large for any context window, the usual approach is retrieval (finding and sending only the relevant sections, the RAG pattern) rather than sending the entire document at once.
Does a larger context window cost more?
Using more tokens costs more regardless of the window's maximum size, since pricing is per token actually sent and generated. A large context window is a ceiling on what's possible, not a cost by itself if a request doesn't use all of it.
Can a conversation ever exceed the context window and still continue?
Only if something actively manages it, summarizing or trimming older turns so the running total stays under the limit. Some platforms offer this as a built-in feature; otherwise an application has to implement that trimming itself.
Interview Questions
What exactly counts toward a context window?
The system prompt, every prior message in the conversation, and the response currently being generated, all counted together against one shared token limit.
What's the difference between a context window and memory?
A context window is temporary and scoped to a single conversation or request; it disappears once that conversation ends. Persistent memory across separate conversations isn't a built-in model feature, it's something an application builds by explicitly storing and re-supplying information later.
Why might a very long context actually hurt answer quality?
Context rot: as token count grows, a model's accuracy and recall degrade, and information buried in the middle of a long context tends to be recalled less reliably than information near the start or end.
What are the two distinct ways a request can fail to fit in a context window?
If the input alone already exceeds the limit, the request is rejected before generation starts. If the input is within limits but generation runs long enough to hit the limit, the response stops early with a specific reason indicating the context window was reached.
Summary
A context window is the shared token budget covering the system prompt, the full conversation so far, and the response being generated. Exceeding it produces an explicit, detectable outcome rather than a silent failure. A bigger window isn't automatically better, since accuracy can degrade as irrelevant content accumulates, and it's fundamentally different from persistent memory, which has to be built deliberately on top of a model, not assumed to exist by default.
What's Next?
The next lesson covers temperature and sampling, the settings that control how deterministic or creative a model's output actually is.