Chunking

Chunking is the first decision you make when building a system that answers questions from your own documents — like a company assistant that reads its manuals, or a lawyer searching case files. Get chunking wrong and no amount of clever model choice will fix it: the evidence is simply not there to retrieve. Get it right and the rest of the pipeline — embeddings, retrieval, generation — has something solid to work with. Mastering chunking unlocks the next steps in retrieval-augmented generation: embeddings, vector databases, and retrieval quality. It also connects directly to your neighbouring notes on admissible evidence spans and citations: if a chunk is too messy, you cannot trace an answer to a defensible passage.

Imagine you have a huge pile of loose papers — your company's entire policy handbook, say. Someone asks you: "What is the leave policy for new mothers?" You cannot hand them the whole pile; they would drown. You need to find the relevant page or paragraph and give them that. But the pile is not organised into helpful sections — it is one continuous stream of text. Chunking means cutting that stream into manageable pieces, each of which could be a coherent answer to some possible question. The size of each piece is a trade-off: cut too small, and you lose the context — a single sentence about 'maternity leave' might not mention the '12 weeks' that appear two sentences later. Cut too large, and you dilute the meaning — a 50-page chapter contains too many topics, so when you search for 'leave policy', the whole chapter matches, but the relevant part is buried. The art of chunking is choosing boundaries that respect the text's own structure — paragraphs, sections, tables — so each piece is self-contained and meaningful.

Think of chunking like cutting up a large pizza for a group of people. You want each person to get a slice that has a bit of crust, a bit of topping, and is easy to hold. If you cut huge slices, some people get mostly cheese, some get mostly crust, and the toppings are uneven. If you cut tiny slivers, each person gets a bite but has to take multiple slices, and the toppings get spread so thin that no one tastes the pepperoni properly. The same with text: a chunk that is too big mixes many topics, so when you search for 'leave policy', the system retrieves a chunk that also talks about 'work-from-home' and 'performance bonuses' — the answer is diluted. A chunk that is too small might only contain 'maternity leave' without the crucial '12 weeks' detail — the answer is incomplete. The best slices are cut along the natural lines of the pizza — the slice separators, which are like paragraph and section boundaries in a document. But the analogy breaks down: pizza slices are all roughly the same size, whereas chunks can vary — a short paragraph might stand alone, while a long section might need to be split. The pizza also has no overlap, but chunks often deliberately overlap so that sentences that straddle a boundary are not lost. So chunking is not a fixed grid; it is a thoughtful division that follows the text's own structure, with some overlap to preserve continuity.

Chunking is the process of splitting a document into smaller, self-contained units — typically a few sentences or a paragraph — so that each unit can be separately indexed, retrieved, and used as evidence for an answer.

You have not yet studied embeddings or vector databases, but chunking is the first stage of a RAG pipeline. Your library notes that the pipeline is 'ingest, chunk, embed, retrieve, rerank, generate' — chunking comes second, right after you bring a document in. It is the highest-leverage choice because every later stage depends on what you feed it. If you chunk poorly, your embeddings (which you will learn later) have nothing meaningful to represent, your retrieval (which you will learn as 'hybrid and semantic search') cannot find the right passages, and your generation (which you know from LLMs) has no grounded evidence to cite. Your neighbouring notes on 'admissible evidence spans' and 'citations' build directly on chunking: to cite a precise passage, the chunk must be small enough to be a defensible unit. And your notes on 'context precision and recall' will show you how to measure whether your chunks are any good.