After a model computes a probability for every possible next token, something still has to pick one. Sampling is that selection step, and temperature is the main dial controlling how it behaves: low temperature makes the model strongly favor the highest-probability token, high temperature makes it far more willing to pick something less likely.
Learning Objectives
- Explain what temperature actually changes in the token-selection process.
- Explain what top-p and top-k restrict, and how they differ from temperature.
- Choose an appropriate temperature for a given kind of task.
Where Temperature Fits in Generation
Raw Probabilities
Temperature Applied
reshapes the curve
Sample
Token Chosen
Temperature is applied before sampling, it reshapes the probability distribution itself. Sampling then picks from that reshaped distribution, it doesn't always take the single highest-scoring token.
What Temperature Actually Does
| Setting | Effect on the distribution | Practical result |
|---|---|---|
| Low (e.g. 0-0.3) | Sharpens it, the top token dominates even more | Focused, repeatable, conservative output |
| Medium (e.g. 0.5-0.8) | Roughly preserves the model's natural distribution | A reasonable balance for general use |
| High (e.g. 1.0+) | Flattens it, lower-probability tokens become viable | More varied, surprising, occasionally incoherent output |
Temperature 0 is often described as "fully deterministic," and it's close, always taking the top token, but some providers' infrastructure (parallel computation, floating-point rounding) can still introduce tiny run-to-run variation even at 0.
Restricting the Pool: Top-k and Top-p
Temperature reshapes probabilities across the entire vocabulary. Top-k and top-p instead restrict which tokens are even eligible to be sampled, before temperature or sampling is applied.
Neither "k" nor "p" is an acronym, both are just placeholder letters for a number you choose. In top-k, k is a whole number (e.g. 40) setting a fixed candidate count. In top-p, p is a probability between 0 and 1 (e.g. 0.9) setting a cumulative-probability threshold instead of a fixed count. Top-p's more descriptive full name is nucleus sampling, from the paper that introduced it.
Full Vocabulary
tens of thousands
Top-k / Top-p Filter
Candidate Pool
much smaller
Sample from Pool
Top-k keeps only the k highest-probability tokens (e.g. top-k 40 keeps exactly 40 candidates, always). Top-p (nucleus sampling) instead keeps the smallest set of tokens whose combined probability reaches p (e.g. top-p 0.9 keeps just enough tokens to cover 90% of the probability mass), so the pool size itself varies request to request depending on how confident the model is.
When to Use Lower vs Higher Settings
- Code generation, where syntax correctness matters more than variety
- Factual Q&A and data extraction
- Anything where the same input should reliably produce a similar output
- Creative writing, brainstorming, generating multiple distinct options
- Casual conversation where some variation feels more natural
- Anything where a wider range of plausible answers is actually the goal
Common Beginner Mistakes
- Raising temperature to "make the model smarter" — temperature only affects token selection randomness, not the model's underlying capability or reasoning.
- Assuming temperature 0 guarantees byte-for-byte identical output on every single run, when minor infrastructure-level nondeterminism can still occur.
- Using a high temperature for a task needing strict correctness (code, structured data) and being surprised by inconsistent or invalid output.
- Combining top-k and top-p without understanding both apply together, the effective pool is whichever restriction ends up smaller.
FAQ
If I want the most reliable, repeatable output possible, what should I set?
Temperature 0 (or very close to it) is the standard choice for maximum consistency, since it makes the model almost always select the single highest-probability token.
Do I need to set both temperature and top-p?
No, most providers recommend adjusting one or the other, not both at once, since they interact in ways that can be hard to predict together. Changing both simultaneously makes it harder to know which setting caused a given change in output.
Does a higher temperature make a model more likely to hallucinate?
It increases the chance of picking a less likely, and therefore potentially less accurate, token at any given step, so yes, practically it tends to increase the rate of factual errors on tasks where precision matters.
Interview Questions
What does the temperature parameter actually control?
It reshapes the probability distribution over possible next tokens before sampling: lower temperature sharpens the distribution toward the highest-probability token, higher temperature flattens it, giving lower-probability tokens a more realistic chance of being picked.
What's the difference between top-k and top-p?
Top-k always keeps a fixed number of the highest-probability tokens as candidates. Top-p keeps a variable number of tokens, whichever smallest set covers a target cumulative probability, so the candidate pool size itself changes based on how confident the model is at that step.
Why might temperature 0 not produce perfectly identical output every single time?
Temperature 0 makes token selection effectively deterministic at the logic level, but underlying infrastructure factors, parallel computation and floating-point rounding differences, can still introduce tiny variations in practice on some systems.
For a code-generation task, would you recommend a low or high temperature, and why?
Low. Code has strict syntactic and logical correctness requirements where variety isn't valuable, a lower temperature favors the model's most confident, typically most correct, token choices.
Summary
Temperature reshapes the probability distribution over next tokens before sampling, controlling how strongly the model favors its top choice versus exploring less likely ones. Top-k and top-p instead restrict the candidate pool itself. Lower settings suit tasks needing consistency and correctness; higher settings suit tasks where variety and creativity are the actual goal.
What's Next?
The next lesson covers prompting basics, how system and user instructions, roles, and constraints actually shape a model's response.