Chunking for RAG: how to split documents so AI can find answers

When people build their first document assistant, they spend most of their time choosing models. Then they discover the answers are mediocre, and that the real lever was something much more mundane: how the documents were cut into pieces. Chunking is one of the most common questions in every community that builds retrieval-augmented generation (RAG) systems, and for good reason. It quietly sets the ceiling on quality.

This article explains what chunking is, what goes wrong when it is done badly, the main strategies, and a method for choosing one without guessing.

Why chunk at all?

A language model can only read so much at once, and a search needs to find specific passages, not entire books. So before a document is indexed, it is divided into chunks, and each chunk is turned into an embedding that can be searched. When someone asks a question, the system retrieves the few best chunks and gives them to the model.

That means the model never sees your whole document. It sees the chunks that retrieval selected, and nothing else. If the right information was cut in half, buried in a huge chunk or stripped of its context, the answer will suffer however good the model is.

What goes wrong at each extreme

• Chunks too large: each one covers several topics, so its embedding becomes a vague average. Retrieval returns bloated passages, wastes the model’s limited space and buries the relevant sentence.
• Chunks too small: a sentence on its own often loses meaning. “It must be submitted within five days” is useless if the chunk does not say what “it” is.
• Chunks cut at random points: a rule is separated from its exception, or a table from its heading, and the model confidently states half the truth.

The main strategies

• Fixed size. Split every N characters or tokens. It is simple and predictable, and a fine place to start, but it ignores the meaning of the text.
• Structure-aware. Split on headings, sections and paragraphs. This follows how the author organised the ideas, and is usually the best default for manuals, policies and reports.
• Sentence or paragraph groups. Combine a few sentences until you reach a target size, never breaking in the middle of a sentence.
• Hierarchical. Keep both small chunks, for precise matching, and the larger section they belong to, which you pass to the model for context.
• Semantic. Split where the topic changes, detected using embeddings. It can be good, but is slower and needs testing.

Things that help whatever strategy you choose

• Overlap. Let neighbouring chunks share a sentence or two, so that an idea at the boundary appears whole in at least one of them.
• Context in every chunk. Prepend the document title and section heading, so that “the deadline is five days” is stored with what it refers to.
• Metadata. Store the document name, date, type, owner and page. You can then filter, prefer recent versions and show precise citations.
• Special handling for tables and lists. Keep a table with its caption and column headings. Convert it to text that preserves the meaning of the rows.
• Clean text first. Remove page headers, footers and repeated navigation text, which pollute chunks and search results.

How big should a chunk be?

Many teams begin with chunks of roughly 200 to 500 words, with an overlap of 10 to 20 percent, and then adjust. Treat these numbers as a starting point, not a rule. Legal clauses may need small, precise chunks. A narrative report may work better with larger ones. The right size depends on your documents and your questions.

A method for choosing, instead of guessing

1. 01Write 30 to 50 real questions, and for each one note the passage that holds the answer.
2. 02Build the index with a first strategy, for example structure-aware chunks of about 300 words with a short overlap.
3. 03Measure retrieval: for what share of questions does the right passage appear in the top few results?
4. 04Change one thing, such as the size, the overlap or adding headings, and measure again.
5. 05Keep the setting that scores best, and look at the questions that still fail to find out why.

Common warning signs

• Answers that are correct but incomplete, because the exception was in another chunk.
• The assistant quoting the wrong version of a document, because dates were not stored.
• Good results on short documents and poor results on long ones.
• Exact codes and names that are never found, which suggests adding keyword search beside embeddings.

Chunking is not a one-time job

When documents change, or when you notice recurring failures, revisit your chunking. Re-index regularly, and keep your test questions as a safety net, so you can see immediately whether a change helped or hurt.

A worked example: a leave policy

Take a policy page with the heading “Annual leave”. It contains three paragraphs: the entitlement of 20 days, the rule that up to 5 unused days may be carried over until March, and the exception that carry-over is not allowed for staff on probation.

• Poor chunking splits the page every 300 characters. One chunk ends with the carry-over rule, and the exception for probation starts the next one. A search about carry-over retrieves only the first, and the assistant says that everyone may carry over 5 days. That is wrong for new staff.
• Better chunking keeps the whole section together, with the heading “Annual leave” added to the front. The chunk contains the rule and its exception, and the assistant answers correctly.

Nothing about the model changed. Only the way the text was cut.

Tables, PDFs and other awkward inputs

• Tables: keep each table with its title and column headings. Consider turning rows into short sentences, such as “Standard plan: 10 users, 20 dollars a month”.
• PDFs: text extraction can scramble columns, and merge headers and footers into the body. Check the extracted text by eye before indexing.
• Scans: run text recognition first, and check the quality, since errors spread into every later step.
• Slides: each slide is short, so combine neighbouring slides with their section title.
• Code and logs: split on functions or entries, not on characters.

Frequently asked questions

What chunk size should I start with?

Many teams start with about 200 to 500 words and 10 to 20 percent overlap, then adjust after measuring retrieval on their own questions.

Should chunk size be measured in words or tokens?

Tokens match what models and embedding services actually count, so use them if your tools support it. Words are a good approximation for planning.

Do I need to re-chunk when I change the embedding model?

You must re-embed everything. You do not have to change the chunk boundaries, but different models may prefer different sizes, so it is worth testing.

Can I use several chunk sizes at once?

Yes. A common approach is to search small chunks for precision and then return the larger section they belong to, so the model sees more context.

Good chunking is unglamorous work, but it often delivers a bigger improvement than switching to a more powerful model. If your assistant is disappointing, look at the chunks first.