Plain similarity search alone often isn't the whole retrieval story in a real system. Real RAG pipelines usually combine several techniques together: narrowing the search with filters, blending semantic and keyword matching, deciding how many results to return, and often re-ordering the results a second time before they ever reach the model.
Learning Objectives
- Explain what metadata filtering and hybrid search each add beyond plain similarity search.
- Explain what top-K controls, and the trade-off in choosing it.
- Explain what a reranking step does and why it's a separate pass from initial retrieval.
A Full Retrieval Pipeline
Query
Similarity Search
Top-K Results
Reranker
Final Context
Each stage narrows or reorders the candidate set. Similarity search casts a reasonably wide net quickly; later stages spend more computation on a much smaller set of candidates to get the final ordering right.
Metadata Filtering
search(queryVector, topK: 5)
search(queryVector, topK: 5, filter: { category: "refunds", region: "US" })Filtering narrows the candidate pool to only entries matching specific structured criteria before similarity even gets applied, useful when a query has a known constraint (a region, a document type, a date range) that similarity alone can't reliably capture.
Hybrid Search: Combining Semantic and Keyword Matching
Pure semantic search can occasionally miss an exact, specific term that matters a lot, a product code, an error message, a proper noun. Hybrid search runs a traditional keyword search alongside the embedding-based similarity search and combines both rankings, catching cases either approach alone would miss.
Semantic Search
meaning-based
Keyword Search
exact terms
Combined Ranking
Choosing Top-K
| Top-K setting | Trade-off |
|---|---|
| Too low (e.g. 1-2) | Risks missing a relevant chunk that scored slightly lower but was still useful |
| Too high (e.g. 50+) | Adds irrelevant context, more tokens, more cost, and can dilute the model's focus |
| A reasonable middle ground (e.g. 3-10) | Balances coverage against noise, often tuned per application |
Reranking: A Second, More Careful Pass
Initial similarity search is optimized for speed across a huge collection, and can be a rough ranking. A reranker takes the smaller top-K set and re-scores each candidate more carefully, often using a more computationally expensive model that directly compares the query against each candidate individually. This is affordable specifically because it only runs on a handful of candidates instead of the entire collection.
Common Beginner Mistakes
- Relying on similarity search alone for queries that need an exact term match, a product code or specific name easily gets lost in purely semantic ranking.
- Setting top-K arbitrarily high, assuming more retrieved context always helps, when it often adds cost and noise instead.
- Skipping reranking entirely on a system where retrieval quality genuinely matters, leaving the rougher initial ranking as the final one.
- Forgetting that metadata filters need the metadata to actually be stored accurately at ingestion time, a missing or wrong field silently breaks filtering later.
FAQ
Is reranking always worth the extra step?
Not for every system, it adds latency and cost. It's most worth it when initial retrieval quality genuinely matters and the small delay is acceptable, less so for a system where speed is the priority and rough relevance is good enough.
Can hybrid search be used without metadata filtering, or do they have to go together?
They're independent techniques and can be used separately or together. Hybrid search blends two ways of scoring relevance; metadata filtering narrows the candidate pool by structured criteria, a system can use either, both, or neither.
Does a higher top-K always improve the final answer?
No, past a certain point additional retrieved chunks add irrelevant context and cost without improving, and can sometimes dilute, the model's ability to focus on what's actually relevant.
Interview Questions
What does metadata filtering add on top of plain similarity search?
It narrows the candidate pool using structured criteria (category, date, region) known ahead of time, something pure vector similarity isn't reliably built to capture on its own.
Why would a system use hybrid search instead of semantic search alone?
Pure semantic search can miss queries where an exact term matters, like a product code or specific name. Hybrid search combines keyword matching with semantic similarity, catching cases either approach alone would miss.
What's the trade-off in choosing a top-K value?
Too low risks missing a relevant chunk that scored just below the cutoff. Too high adds irrelevant context, more token cost, and can dilute the model's focus on what actually matters.
Why is reranking typically a separate step from initial retrieval, rather than doing everything in one pass?
Reranking uses a more computationally expensive comparison than initial similarity search, which is only affordable because it runs on a small top-K set of candidates, not the entire collection. Doing that expensive comparison against every stored vector directly wouldn't scale.
Summary
A real retrieval pipeline often combines several techniques: metadata filtering to narrow by structured criteria, hybrid search to catch exact-term matches semantic search alone might miss, a tuned top-K to balance coverage against noise, and reranking as a more careful second pass over a small candidate set. Together these push retrieval quality well beyond what plain similarity search alone achieves.
What's Next?
The next section covers MCP in depth, starting with what the protocol actually is and why it exists.