A single embedding is meant to represent one coherent idea. A 50-page document embedded as one vector loses almost all of its specific detail, the vector ends up representing a vague average of everything in it. Chunking splits a document into smaller pieces first, each embedded and stored separately, so retrieval can find the one specific section that actually answers a question.
Learning Objectives
- Explain why documents get split into chunks before embedding.
- Compare fixed-size chunking against semantic chunking.
- Explain what chunk size and overlap each control, and the trade-offs of adjusting them.
Document to Chunks to Embeddings
Full Document
Split into Chunks
Embed Each Chunk
Store in Vector DB
Each chunk becomes its own independent, searchable unit. A query only needs to match one relevant chunk to succeed, it doesn't need to match the entire original document.
Fixed-Size vs Semantic Chunking
- Splits every N tokens or characters, regardless of content
- Simple and fast to implement
- Can cut a sentence or idea awkwardly in half
- Splits at natural boundaries: paragraphs, sections, topic changes
- More setup, sometimes needs its own model to detect boundaries
- Each chunk more reliably represents one coherent idea
Chunk Size and Overlap
Overlap means adjacent chunks share a small amount of text at their boundary, so an idea that happens to fall right at a chunk boundary doesn't get split with no chunk containing the complete thought. Too little overlap risks losing context at boundaries; too much overlap means storing and searching redundant text.
Chunk 1
Overlap Region
shared text
Chunk 2
Bad vs Good Chunking, Concretely
Chunk 1: "...the refund window is 30 days from the" Chunk 2: "date of purchase, excluding final sale items..."
Chunk 1: "...the refund window is 30 days from the date of purchase." Chunk 2: "The refund window is 30 days from the date of purchase, excluding final sale items."
In the good example, either chunk alone still contains a complete, usable statement of the policy, retrieval doesn't depend on getting both halves back together correctly.
Common Beginner Mistakes
- Choosing an arbitrarily small chunk size that splits a single idea across several fragments, hurting retrieval quality.
- Choosing an oversized chunk that dilutes the specific detail a query is actually looking for, back to the whole-document problem chunking was meant to avoid.
- Using zero overlap and being surprised when a fact sitting right at a chunk boundary never gets retrieved cleanly.
- Applying the same fixed chunk size uniformly to very differently structured documents (a legal contract vs a chat transcript) without adjusting the approach.
FAQ
Is there one universally correct chunk size?
No, it depends on the content and the kind of questions expected. Dense technical documentation often benefits from smaller chunks; narrative or conversational content often works better with larger ones. Testing against real queries is more reliable than picking a number in the abstract.
Does chunking affect cost, not just retrieval quality?
Yes, more chunks means more embeddings to compute and store, and a chunk that gets retrieved becomes part of the prompt, so its size directly affects token cost on every request that retrieves it.
Can headings and document structure be used to guide chunking?
Yes, this is a common and effective form of semantic chunking, splitting along a document's own headings or sections rather than an arbitrary character count, since the author already marked where one idea ends and another begins.
Interview Questions
Why is a single embedding for an entire long document usually a bad idea?
One vector ends up representing a vague average of everything in the document, losing the specific detail needed to match a narrow, specific query. Chunking lets each smaller, focused piece be embedded and matched independently.
What's the trade-off between fixed-size and semantic chunking?
Fixed-size chunking is simple and fast but can cut an idea awkwardly across a boundary. Semantic chunking respects natural content boundaries, producing more coherent chunks, at the cost of more implementation complexity.
What problem does overlap between chunks solve?
Without overlap, an idea sitting right at a chunk boundary can end up split, with neither chunk containing the complete thought. A small overlap ensures boundary content is still fully represented in at least one chunk.
How does chunk size affect token cost, not just retrieval quality?
A retrieved chunk becomes part of the prompt sent to the model, so larger chunks mean more tokens consumed per retrieved result, directly increasing the cost of every request that retrieves them.
Summary
Chunking splits a document into smaller, independently searchable pieces before embedding, since a single vector for an entire long document loses too much specific detail. Semantic chunking respects natural content boundaries better than fixed-size splitting, and a small overlap between chunks prevents an idea from being cut apart at a boundary with no single chunk containing it whole.
What's Next?
The next lesson covers retrieval strategies, how a system actually decides which chunks to return for a given query, beyond plain similarity search.