An LLM (Large Language Model) is a model trained to predict the next word in a sequence of text, given everything that came before it. That single mechanism, repeated one word at a time, is what produces everything from a one-line answer to a full explanation. There's no separate "understanding" module and no built-in fact database, the entire system is one very large prediction function.
Learning Objectives
- Explain what an LLM actually does at inference time, in concrete terms.
- Distinguish training from inference, and explain why that distinction matters.
- Explain how an LLM differs from traditional, rule-based software.
The Core Idea: Next-Token Prediction
Given the text "The capital of France is", an LLM doesn't look up an answer in a table. It calculates a probability for every possible next token in its vocabulary, and "Paris" scores highest because of patterns learned from massive amounts of text during training. It picks a token, appends it, and repeats the entire calculation for the next one.
The Inference Flow
Input Text
Tokenizer
text → tokens
Model
transformer layers
Probabilities
over the vocabulary
Output Token
This entire flow runs once per output token, not once per response. A 200-token answer means this exact sequence ran roughly 200 times, each time with the previously generated token appended to the input.
Training vs Inference
| Training | Inference | |
|---|---|---|
| What happens | The model's internal parameters are adjusted based on massive amounts of text | The already-trained model generates output for a specific input |
| Frequency | Happens once (or occasionally, for a new version) | Happens every single time someone sends a request |
| Cost | Extremely expensive, can take weeks on large hardware clusters | Comparatively cheap, but still real per-request cost |
| What a developer controls | Nothing, unless fine-tuning a model | The prompt, and parameters like temperature |
Massive Text Data
Adjust Parameters
repeated, millions of times
Trained Model
happens once
This distinction matters constantly in practice: a wrong or outdated answer almost always means a context problem at inference time (missing information in the prompt), not something to fix by "retraining," which is a separate, far more expensive undertaking.
How This Differs From Traditional Software
- Explicit rules written by a developer (if X, then Y)
- Deterministic: the same input always produces the same output
- Behavior changes only when the code changes
- Errors are usually reproducible and traceable to a specific line
- Behavior emerges from patterns learned across training data, not explicit rules
- Probabilistic by default: the same input can produce different output on different runs
- Behavior can shift with a different prompt, with no code change at all
- A wrong answer often has no single traceable "bug", it's a statistical outcome
Real Developer Use Cases
- Summarizing a long document into a few key points
- Converting a natural-language request into structured data (JSON, a database query)
- Drafting, explaining, or reviewing code
- Classifying free-text input (support tickets, reviews) into categories
Advantages and Limitations
- + Handles unstructured, messy natural-language input that traditional rule-based code struggles with
- + One general-purpose model covers a huge range of tasks without task-specific programming
- + Improves with better prompting alone, no redeployment needed
- − Can produce confident, fluent, and factually wrong output (hallucination)
- − No built-in awareness of anything after its training cutoff, or of private data it was never shown
- − Output isn't strictly deterministic, which complicates testing and debugging
- − Real per-request cost that scales with usage, unlike a one-time software cost
Common Beginner Mistakes
- Assuming an LLM "looks things up" the way a search engine does, it predicts text based on learned patterns, it doesn't query a live source unless something like RAG explicitly gives it one.
- Expecting identical output on every run without setting parameters (like temperature) that control randomness.
- Treating a wrong answer as a bug to be patched in the model, when the actual fix is almost always a better prompt or better supplied context.
- Assuming a bigger model is always the right choice, latency and cost scale with size too, and many tasks don't need the largest available model.
FAQ
Does an LLM actually "know" anything, or is it just predicting words?
Mechanically, it's predicting the next token based on learned statistical patterns. Whether that constitutes "knowing" is a genuinely debated question, but practically, the useful framing is: it's very good at producing plausible, pattern-consistent text, which is a different guarantee than having verified facts.
Why do LLMs sometimes give a different answer to the exact same question?
Generation involves sampling from a probability distribution over possible next tokens, not always picking the single highest-scoring one. That controlled randomness (adjustable via temperature) is why output isn't always identical run to run.
What's the difference between an LLM and a chatbot?
An LLM is the underlying model. A chatbot is an application built around one, adding a conversation interface, message history handling, and usually additional logic (safety filters, tool access) on top of raw next-token prediction.
Interview Questions
In one sentence, what does an LLM actually compute?
Given a sequence of tokens, it computes a probability distribution over every possible next token, and generation is the repeated process of sampling from that distribution one token at a time.
Why can't a wrong answer from an LLM always be fixed by "retraining" it?
Retraining changes the model's parameters globally and is extremely expensive; it's not a targeted fix for one bad answer. Most wrong answers are actually a context or prompting problem at inference time, addressed by supplying better information in the prompt, not by retraining.
Why is LLM output not fully deterministic by default?
Generation samples from a probability distribution rather than always selecting the single most likely token, controlled by parameters like temperature. Setting temperature to 0 makes output far more consistent, though some sampling systems retain minor nondeterminism even then.
What's a concrete example of a task well-suited to an LLM versus one that isn't?
Well-suited: classifying free-text customer feedback into categories, since the input is unstructured and language-heavy. Poorly suited: computing an exact tax calculation, since that needs guaranteed, deterministic arithmetic, not a probabilistic text prediction.
Summary
An LLM predicts the next token in a sequence, one token at a time, based on patterns learned during an expensive, one-time training process. That single mechanism is probabilistic rather than rule-based, which explains both its flexibility with messy natural language and its real limitations: it can be confidently wrong, and it knows nothing beyond its training data and whatever context it's given at request time.
What's Next?
The next lesson covers tokens in more depth, exactly what gets counted toward a model's limits and API cost, and why a word isn't always one token.