A language model's knowledge is frozen at training time. Ask it about something that happened after that cutoff, or about a private document it was never shown, and it either says it doesn't know or, worse, generates a confident, plausible-sounding answer that's simply wrong. RAG (Retrieval-Augmented Generation) exists specifically to close that gap, by searching real, current information and handing it to the model before it answers.
Learning Objectives
- Explain the specific limitation of LLMs that RAG addresses.
- Distinguish RAG from fine-tuning, and explain when each is the right tool.
- Recognize real-world scenarios that genuinely call for RAG.
The Problem: A Frozen Snapshot of Knowledge
Model Trained
on a fixed dataset
Training Cutoff
New Question Asked
after the cutoff
Guess or Wrong Answer
This isn't a bug that gets fixed with a better model, it's a structural property of how training works. Every model, no matter how capable, has some point after which it simply has no information at all, and no way to know that it's missing something.
RAG vs Fine-Tuning: Two Different Fixes
- Adjusts the model's actual weights through additional training
- Slow and resource-intensive, not something done per request
- Good for teaching a new skill, style, or behavior pattern
- Updating knowledge means retraining again, a real ongoing cost
- Changes nothing about the model itself
- Supplies fresh, relevant text at the moment of answering
- Good for supplying current facts or private, domain-specific data
- Updating knowledge means editing a document, near-instant
These solve different problems and are often used together: fine-tuning to change how a model behaves or writes, RAG to change what facts it has access to when answering.
Real Scenarios That Actually Need RAG
- An internal support bot answering questions from a company's own private wiki, something no public model was ever trained on.
- A legal or compliance assistant that must cite the current version of a specific regulation, not whatever version existed at training time.
- A product documentation assistant that needs to reflect a feature shipped last week.
- Any assistant where a wrong, unsourced answer is a genuinely costly mistake, not just an inconvenience.
Question Asked
Search Real Documents
Relevant Text Found
Grounded Answer
When RAG Is Overkill
Not every task needs it. A request that only needs general reasoning, common knowledge well within the model's training data, or creative generation with no factual grounding requirement, gets no benefit from adding a retrieval step, just added latency and infrastructure for no real gain.
Common Beginner Mistakes
- Reaching for RAG as a default for every project, even when the task doesn't actually involve any private or time-sensitive information.
- Expecting RAG to fix a model's reasoning ability, it improves what facts the model has access to, not how well it reasons about them.
- Confusing RAG with fine-tuning, and choosing the slower, more expensive option to solve a problem retrieval would have handled directly.
- Assuming RAG guarantees a correct answer, a poor retrieval step still produces a confidently wrong one, just now grounded in the wrong source instead of memory.
FAQ
Can RAG and fine-tuning be used in the same system?
Yes, and it's common. Fine-tuning to adjust tone, style, or a specialized skill, RAG to supply current or private facts the fine-tuned model still wouldn't otherwise know.
Does RAG make a smaller, cheaper model perform as well as a larger one?
Not universally, but on knowledge-heavy tasks specifically, a smaller model with good retrieval can outperform a larger model working from memory alone, since the smaller model isn't guessing, it's reading.
Is RAG only useful for text documents?
No, the same retrieve-then-generate pattern applies to any searchable source, structured database records, code repositories, or transcripts, as long as it can be converted into a searchable form.
Interview Questions
What specific limitation of LLMs does RAG address?
A model's knowledge is fixed at training time. RAG doesn't change the model, it searches current, external information and supplies it as context before the model generates an answer, sidestepping the frozen-knowledge problem entirely.
Why would you choose RAG over fine-tuning to add new information to a system?
Fine-tuning requires retraining to update, is slow and resource-intensive. RAG only requires updating the underlying documents, which is close to instant and far cheaper, making it the better fit when the actual need is fresher or private facts rather than a different behavior or style.
Give an example of a task where adding RAG would provide no real benefit.
A general creative writing request, or a question well within common knowledge the model was already trained on. Retrieval adds latency and infrastructure cost without improving an answer that didn't have a knowledge gap to begin with.
Does RAG guarantee accurate answers?
No. It grounds the answer in retrieved text instead of memory, which reduces one source of error, but a retrieval step that returns irrelevant or low-quality results still produces a confidently wrong answer, just built from the wrong source.
Summary
RAG exists to solve a structural limitation of LLMs: their knowledge is frozen at training time. Rather than retraining the model (fine-tuning), RAG retrieves current or private information at request time and supplies it as context. It's the right tool when a system needs fresh or private facts, not when the actual need is a different behavior, style, or purely creative output.
What's Next?
The next lesson covers embeddings, the technique that converts text into a form that can be searched by meaning instead of exact wording.