Why your AI chat sometimes "forgets"
You're forty messages into a long conversation with an AI assistant. You mentioned an important detail near the start, something you're relying on it to remember, and now it's acting like it never saw it. That's not the model being careless. It's two specific, mechanical limits: tokens and the context window. Once you know what those actually are, the behavior stops feeling random.
What a token actually is
A token is the actual unit an AI model reads and generates text in, and it's neither a word nor a character. Models use subword tokenization: common whole words often become one token, longer or less common words get split into a few meaningful pieces, and unusual strings (a typo, a rare name, a chunk of code) can take more tokens per character than plain English prose does.
A rough, commonly cited rule of thumb for English text: about 4 characters per token, or roughly 100 tokens per 75 words. It's an approximation, not a formula, code, non-English text, and unusual strings all break it in different directions.
| Text | Tokens (approximate) | Why |
|---|---|---|
| "the" | 1 | Common whole word, its own token |
| "tokenization" | 2-3 | Split into meaningful word pieces |
| "xk7#pQz!" | 4-6 | No recognizable pattern, falls back toward individual characters |
| A 10-line Python function | ~80-150 | Punctuation, indentation, and symbols each cost tokens too |
Exact counts depend on the specific model's tokenizer, different AI companies use different tokenization schemes, so the same sentence can be a different number of tokens on Claude versus GPT versus Gemini.
What a context window actually is
The context window is the maximum number of tokens a model can hold in view at once, for everything: your messages, its own previous replies, any system instructions running in the background, and the results of any tool it called. It's usually described in a model's documentation as one number, like "200K tokens" or "1M tokens," and that number is a hard ceiling, not a soft suggestion.
This is the part people usually miss: the context window isn't just "how much you've typed." A single long document pasted into the chat, a big block of code the model read from your project, or a long tool result (like a web search's raw output) all count against that same shared budget, right alongside the actual conversation.
Your Messages
Every turn, both sides
System Instructions
Running quietly in the background
Tool Results
Search results, file contents, API responses
One Shared Budget
All of it counts against the same token limit
Context window sizes, in general terms
Specific model-by-model numbers go stale fast, these change with nearly every release, and even reputable sources disagree on the exact current figure for a given model at any given moment. What's stable enough to say plainly: as of late 2026, the major AI assistant families (Claude, GPT, Gemini) have all moved their flagship models toward very large context windows, commonly in the hundreds of thousands of tokens, with their top-tier models reaching roughly a million tokens or more.
For the real, current number on a specific model you're actually building with, check that provider's own docs rather than trusting a cached figure, including this one. A few that stay current:
Why "just make it bigger" isn't free
- + A larger context window means less need to summarize or chunk a long document before handing it to the model
- + It lets a coding agent hold more of a real codebase in view at once, which is part of why modern AI coding tools got noticeably more capable
- + Long, multi-turn conversations stay coherent for longer before earlier details fall out of view
- − Cost: most providers charge per token, a bigger context window means a bigger bill if you actually fill it, every single request
- − Latency: more tokens to process generally means a slower response, not just a cheaper or more expensive one
- − The "lost in the middle" effect: models are measurably more reliable at using information near the start or end of a long context than something buried in the middle of it, a bigger window doesn't fully fix this on its own
This is exactly why retrieval (covered in our MCP vs RAG explainer) usually beats "just paste the whole document in": fetching only the relevant few paragraphs keeps the model's attention on what actually matters, instead of diluting it across thousands of irrelevant tokens.
What actually happens when a conversation runs out of room
Different products handle this differently, and it's worth knowing the shape of the real options, since "it just forgets" undersells what's actually happening:
- Hard cutoff: the oldest messages simply get dropped from what the model can see, once the window is full, the conversation effectively starts losing its earliest turns.
- Automatic summarization: some tools compress older parts of the conversation into a shorter summary before they'd otherwise be dropped entirely, preserving the gist while freeing up token budget. This is a real, common pattern, not a hypothetical, Claude Code itself does exactly this as a long session's context fills up.
- A fresh start: some tools simply won't go further and ask you to start a new conversation once the limit is reached, rather than silently dropping or summarizing anything.
None of these options give the model perfect, permanent recall of everything you've ever said. If a detail genuinely matters later in a long conversation, restating it costs a handful of tokens and is far more reliable than hoping it's still "in view."
Context window vs memory, a genuinely common confusion
A context window is not the same thing as memory. The context window is scoped to one conversation (or one coding session), and once that conversation ends, everything in it is gone unless the product specifically built a separate memory feature on top. Memory, where it exists, is a different system: it deliberately saves specific facts outside the conversation itself and re-injects them into a future context window when relevant, rather than just being more room in the current one.
Put differently: a bigger context window lets today's conversation hold more at once. Memory is what lets tomorrow's conversation know anything about today's at all. They solve different problems, and a product can have either one without the other.
What this actually means if you're building with AI
- Don't paste an entire 300-page document in and hope for the best, chunk it or retrieve only the relevant sections (RAG) instead, you'll get more reliable answers and pay for fewer wasted tokens.
- Put the most important instructions near the start or the end of a long prompt, not buried in the middle, where the "lost in the middle" effect hits hardest.
- Watch your output token limit separately from your context window, most providers cap how long a single reply can be, independent of how much input context you've used.
- If a long-running conversation or agent session needs to remember something specific much later, restate it explicitly near where it's needed rather than trusting it survived a context compression or summarization step untouched.
- When estimating cost, remember tool results and retrieved documents count as real input tokens too, a request with a big search result attached isn't free just because you typed a short question.
Interview Questions
A few questions on this topic that come up in real conversations about working with AI tools, not just textbook trivia.
Is a token the same thing as a word?
No. A token is a subword unit a model's tokenizer produces, a common short word is often one token, but a longer or unusual word can split into two or more tokens, and some short strings of punctuation or symbols can each cost their own token too. The rough rule of thumb (about 4 characters per token for English text) is an approximation, not an exact conversion.
What actually counts against a model's context window?
Everything the model has to read to generate its next response: every message in the conversation so far on both sides, any system instructions running quietly in the background, and the full content of anything a tool call returned (search results, file contents, API responses). It's one shared token budget across all of that, not a separate allowance per category.
Why doesn't a bigger context window just solve long-conversation problems entirely?
Two real limits remain even with a huge window: cost and latency scale with how many tokens you actually use, and models are measurably less reliable at using information buried in the middle of a long context than information near the start or end of it, the "lost in the middle" effect. A bigger window raises the ceiling, it doesn't guarantee every token in it gets used equally well.
What's the real difference between a context window and an AI memory feature?
A context window is scoped to one conversation or session, and everything in it disappears once that conversation ends. A memory feature is a separate system that deliberately saves specific facts outside any one conversation and re-injects them into a future context window when relevant. A product can have a large context window with zero memory, or a small context window with memory layered on top, they're independent capabilities.
Why is retrieval (RAG) often better than just pasting a whole document into the context window?
Pasting an entire long document spends a large chunk of the token budget on mostly-irrelevant content, raises cost, and spreads the model's attention thin, worsening the lost-in-the-middle effect. Retrieving only the specific passages relevant to the current question keeps the useful signal concentrated in a much smaller number of tokens, which tends to produce both cheaper and more accurate answers.
If a long AI conversation needs to remember an important detail from much earlier, what's the reliable fix?
Restate it explicitly, near where it's actually needed. Once a conversation has run long enough to risk hitting a context limit, an earlier detail may already have been dropped, summarized, or diluted by everything said since. A short restatement costs a handful of tokens and is far more reliable than assuming the model still has perfect access to something said many turns ago.
The short version
A token is the actual unit of text a model reads and writes in, not a word and not a character. A context window is the hard limit on how many tokens it can hold in view at once, for the whole conversation, system instructions, and tool results combined, not just what you typed. Bigger windows help, but they cost more, run slower, and don't fully fix a model's weaker grip on information buried in the middle of a long context. And none of it is memory: once a conversation ends, what was in its context window is gone unless a separate memory feature saved it on purpose.