Every LLM reads and writes text as tokens, not as words and not as individual characters. A token is a chunk of text, sometimes a whole word, sometimes part of one, that the model's tokenizer converts into a number before anything else happens. The model never actually sees the string "hello", it sees an integer that stands in for it.
Learning Objectives
- Explain what a token is, and why it isn't the same unit as a word.
- Estimate a rough token count for a piece of text.
- Explain why both input and output tokens affect cost and context limits.
The Tokenization Flow
Raw Text
Tokenizer
splits into chunks
Token IDs
each chunk → a number
Model Input
This runs in reverse on the way out: the model produces token IDs one at a time, and the tokenizer converts each one back into readable text before it's shown to the user.
A Concrete Example
"unbelievable"
1 word
un
believ
able
3 tokens
"I love unbelievable stories." → ["I", " love", " un", "believ", "able", " stories", "."] 7 tokens for 5 words.
Common short words ("I", "love") tend to be a single token each. Less common or compound words ("unbelievable") often split into several pieces, here un / believ / able. Punctuation and even leading spaces can count as their own tokens too.
A Rough Estimate
| Unit | Approximate token count |
|---|---|
| 1 token | ≈ 4 characters of English text |
| 100 tokens | ≈ 75 words of English text |
| 1 page (~500 words) | ≈ 650-700 tokens |
This is a rule of thumb for English specifically, not a fixed rule. Other languages, especially ones that don't use spaces to separate words, commonly tokenize far less efficiently, sometimes needing two to three times as many tokens for the same sentence's meaning.
Why Tokens Aren't Words: the Practical Impact
- A 300-word prompt would need a limit measured in words
- Cost would scale with word count
- The same word would always cost the same regardless of complexity
- Context limits and API pricing are both measured in tokens, not words
- Rare words, code, and non-English text can cost noticeably more per word than plain English
- A model's actual vocabulary is finite (tens of thousands of possible tokens), which is what tokenization is built to fit text into
Tokens and Cost: Both Directions Count
API pricing for an LLM is charged per token, and critically, on both the input (the prompt sent) and the output (the response generated), usually at different rates. A long, detailed system prompt sent on every single request adds real, recurring token cost, not a one-time cost.
The Other Cost: Water and Energy
Every token processed also has a physical cost, not just a billed one. Running the hardware behind a request uses electricity, and a meaningful amount of that electricity, plus the direct cooling systems in the data center itself, uses water.
| Source | Figure | What it represents |
|---|---|---|
| OpenAI (public statement) | ~0.3 mL per query | Direct operational water only, not the full lifecycle |
| Google (comprehensive methodology) | 0.26 mL + 0.24 Wh per median Gemini prompt | Full system: active + idle machines, CPU/RAM, data center overhead |
| Google (active-machine-only methodology) | 0.12 mL per median Gemini prompt | Only active chip usage, Google itself calls this an underestimate |
| Microsoft (infrastructure-level, not per-query) | 125M+ liters saved per facility per year | Newer direct-to-chip, closed-loop cooling design avoiding evaporation entirely |
| UC Riverside research paper (training, not inference) | ~700,000 liters | Water directly evaporated training GPT-3 in Microsoft's US data centers |
Tokens Processed
Compute + Electricity
Data Center Cooling
uses water directly
Water + Energy Footprint
Even a single company can report more than one number for the same thing: Google publishes both an active-machine-only figure and a comprehensive one, and says outright that the narrower figure underestimates real-world usage. Microsoft, rather than a per-query number, reports facility-level savings from newer cooling designs. The right way to state any of this is "X has publicly reported approximately Y, measured as Z", not a bare, unqualified number, since the measurement boundary changes the answer as much as the underlying technology does. These figures come from company disclosures and independent research, not a single agreed-upon standard, so treat them as informative estimates rather than fixed, permanent facts. They can change quickly, even day to day, as hardware, cooling methods, and measurement methodology keep evolving, so always check for the latest published numbers rather than assuming these stay accurate indefinitely.
How to Actually Reduce Token Usage
- Trim conversation history instead of resending every prior turn in full, older turns can often be summarized into a few sentences without losing what's actually needed.
- Use prompt caching where the provider supports it, a large, unchanging system prompt or reference document gets cached once and billed at a reduced rate on later requests, instead of full price every time.
- Retrieve only the relevant section of a large document (the RAG pattern) rather than pasting the entire document into every prompt.
- Set an explicit, sensible output length limit instead of leaving generation open-ended, an unexpectedly long response is billed like any other output tokens.
- Keep system prompts and instructions concise and specific, a shorter prompt that says exactly what's needed usually outperforms a long one padded with restatement.
- Use a smaller, cheaper model for simple, well-defined tasks (classification, short extraction), and reserve a larger model for genuinely complex reasoning.
Full Context
history + docs + prompt
Trim History
Cache the Prompt
reused prefix
Retrieve Only What's Needed
RAG instead of full docs
Lower Token Bill
None of these change what a token is, they reduce how many actually get sent or generated, which is the only thing that affects cost and context usage.
Common Beginner Mistakes
- Assuming word count and token count are interchangeable when estimating cost or checking a limit.
- Forgetting that a model's generated response also counts against both the context window and the bill, not just the prompt sent to it.
- Not accounting for a large system prompt or conversation history being resent (and re-billed) on every single turn of a conversation.
- Assuming all languages tokenize with similar efficiency, when non-English text frequently uses more tokens for the same content.
FAQ
Is a token always part of a word, never a whole sentence or a single letter?
Neither extreme is guaranteed. Common short words often are exactly one token. Rare words split into multiple sub-word tokens. Individual characters and punctuation can each be their own token too. There's no fixed size, it depends entirely on how common that exact chunk of text was in the tokenizer's training data.
Do spaces count as tokens?
Often, yes, indirectly. Many tokenizers attach a leading space to the following word as part of the same token (like " love" in the earlier example), rather than treating whitespace as entirely free.
Can I know the exact token count before sending a request?
Yes, model providers publish the exact tokenizer used, and offer tools (like OpenAI's public tokenizer page) to count precisely, rather than relying on the word-count approximation.
Interview Questions
What is a token, precisely?
A chunk of text, determined by a specific tokenizer, that gets mapped to a single integer ID before being processed by a model. It can be a whole word, part of a word, a punctuation mark, or occasionally a single character.
Why can't token count be reliably estimated as "one token per word"?
Because tokenizers commonly split uncommon or compound words into multiple sub-word tokens, while very common short words are usually exactly one token. The actual ratio varies by content and by language.
Why does a system prompt add ongoing cost, not just a one-time cost?
If that system prompt is resent with every request (which it typically is, since the model has no persistent memory between separate API calls), its tokens are billed again on every single request, not once.
Why might the same sentence cost more in one language than another?
Tokenizers are built primarily around the training data's dominant language, usually English. Languages that don't tokenize as efficiently against that vocabulary need more tokens to represent the same content, directly increasing cost and consuming more of the context window for equivalent meaning.
Summary
A token is the actual unit an LLM reads and writes, a chunk of text mapped to a number, not a word. Roughly 4 characters or 0.75 words per token is a workable English estimate, but actual counts vary by content and language. Both the prompt sent and the response generated consume tokens, which is why token count, not word count, is what genuinely determines cost and context limits.
What's Next?
The next lesson covers context windows, what happens once total token count (prompt plus conversation history plus response) hits a model's limit.