8 · Chunking documents
You can't embed a whole book as one vector. Chunking splits documents into retrievable pieces — and the chunk size and overlap decide whether retrieval is precise or loses the thread.
Before you can retrieve, you must split documents into chunks small enough to embed and to fit the context window, yet large enough to be self-contained. Fixed-size chunks are simple but cut sentences mid-thought; overlap copies a few tokens across the boundary so context survives the cut; sentence-aware splitting respects natural breaks.
Without this:
Chunk too big and each vector is a blurry average — retrieval can't tell which part is relevant. Chunk too small with no overlap and you slice the answer in half, so neither piece contains the full fact. Bad chunking quietly caps the quality of every downstream RAG answer.
Real documents are long — a manual, a PDF, a wiki page. You can't embed a 50-page document as one vector: the meaning would be averaged into mush, and retrieval could never point at the specific paragraph that answers a question. So step zero of every RAG pipeline is chunking: splitting each document into smaller, retrievable pieces, then embedding and indexing those.
Three things pull the chunk size in opposite directions:
- Big enough to be self-contained. A chunk must carry enough surrounding context that, read alone, it still makes sense.
- Small enough to be precise. Smaller chunks give sharper retrieval — the matched vector is about one idea, not a paragraph of mixed topics.
- Small enough to fit the budget. You'll stuff several chunks into the prompt, so each must leave room for the rest plus the answer (lesson 2's token budget).
The classic strategies:
- Fixed-size — every
sizecharacters (or words) becomes a chunk. Dead simple, but it happily cuts a sentence — even a word — in half. - Fixed-size with overlap — each chunk repeats the last
overlapunits of the previous one. The repeated text means a fact straddling a boundary appears whole in at least one chunk. - Sentence- / structure-aware — split on sentence or paragraph boundaries so chunks never cut mid-thought (often combined with a size cap).
Let's implement the first two and watch the overlap save a fact that a hard cut would have severed.
Python (in browser)
chunk_text slides a window of `size` words stepping by `size - overlap`. With overlap, boundary-straddling facts reappear intact in a neighbouring chunk.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Fixed windows are blind to meaning — they'll cut "30 days of purchase" right after "30". Sentence-aware splitting fixes that by breaking only at sentence ends, then packing sentences into chunks up to a size cap. Each chunk is then guaranteed to contain whole sentences, so no fact is ever severed mid-clause.
Python (in browser)
Sentence-aware chunking packs whole sentences up to a word cap, so retrieval always matches against grammatically complete, self-contained facts.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Production splitters (LangChain's RecursiveCharacterTextSplitter) combine all three ideas: prefer natural separators, cap the size, and overlap across boundaries.
Why add overlap between consecutive chunks?
- Chunking splits long documents into retrievable pieces before embedding — too big blurs meaning, too small loses self-containment.
- Overlap repeats text across boundaries so a fact spanning a cut still appears whole in a neighbouring chunk — at the cost of a larger index.
- Sentence- or structure-aware splitting respects natural breaks so chunks never cut mid-sentence; production splitters combine all three strategies.
Every RAG ingestion job — indexing docs, PDFs, code, transcripts — runs a chunker first; chunk size and overlap are the most-tuned knobs when retrieval quality is poor.
If you remove it: Skip chunking and you either can't index long docs at all or you index them as one blurry vector — retrieval returns vague, mixed-topic blobs that don't answer the question.