Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 ยท Chunking and Ingestion

Chunking Strategies

Learn how to split documents into chunks for RAG: fixed-size, overlap, sentence and recursive splitting, how chunk size changes retrieval, and how to choose and test a strategy.

A language model cannot read your whole knowledge base, and an embedding model cannot represent a 40-page manual in one vector without blurring everything together. So documents are cut into chunks, each embedded and stored separately. Chunking is one of the most influential decisions in a RAG system, and one of the least glamorous.

The trade-off is easy to state. Small chunks are precise: a vector about one idea matches questions about that idea, and the prompt stays short. But they lose context ("it expires after 1 hour" - what expires?). Large chunks keep context but dilute the embedding with several topics, match fewer specific questions, and waste prompt tokens on irrelevant text. This lesson shows the standard splitting methods and how to compare them with a test instead of folklore.

Fixed-size chunks with overlap

The simplest splitter cuts every size characters (or tokens) and starts each new chunk overlap characters before the previous one ended. Overlap is insurance: a sentence cut at a boundary still appears whole in the neighboring chunk.

Python Fixed-size splitting, with and without overlap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without overlap, the chunk boundary lands in the middle of a sentence and the fact "restored by an admin within 7 days" is split across two vectors, so neither matches a question about restoring well. With overlap, the second chunk repeats the end of the first, so the sentence survives whole in at least one chunk. The cost: indexed text grows by overlap / (size - overlap) (about 50% here), and neighboring chunks may both be retrieved, repeating text in the prompt.

Typical starting values: 200 to 500 tokens per chunk with 10% to 20% overlap. The right size depends on your content and questions: FAQs with short answers like small chunks; legal or technical explanations like larger ones.

Characters, words or tokens?

Embedding models and LLMs limit tokens, not characters. A token is roughly 3/4 of an English word (less efficient for other scripts and for code). If a chunk exceeds the embedding model's limit (model limits), its end is silently dropped. Measure chunk size with the model's own tokenizer when you can; word counts are a fair approximation for planning.

Sentence-based chunks

Cutting mid-sentence is the main flaw of fixed-size splitting. A sentence-based splitter first finds sentence boundaries, then packs whole sentences into a chunk until a size budget is reached. Sentence detection can be a simple regular expression; for production text with abbreviations ("Dr.", "e.g.") use a proper sentence tokenizer.

Recursive splitting

The most common general-purpose strategy is recursive: try the biggest natural separator first (blank lines between paragraphs), and only if a piece is still too big, split it by the next separator (single newlines, then sentences, then words, finally characters). It respects document structure as far as the size budget allows.

flowchart TD
    A["Text"] --> B{"Fits in budget?"}
    B -->|yes| C["Emit one chunk"]
    B -->|no| D["Split on paragraph breaks"]
    D --> E{"Piece still too big?"}
    E -->|yes| F["Split on sentences, then words"]
    E -->|no| G["Merge neighboring small pieces up to the budget"]
    F --> G
    G --> C

Recursive splitting: big separators first, merge small pieces back up to the budget.

Python A recursive splitter with a word budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each chunk is a clean unit made of whole sentences, never longer than the budget. The is_heading check keeps a section heading with the text that follows it instead of leaving it stranded at the end of the previous chunk. A refinement worth knowing is to keep the heading with every chunk of its section (structure-aware chunking).

Measuring the effect of chunk size

Opinions about the best chunk size are cheap; data is better. The experiment below builds a small knowledge base, splits it at several sizes, embeds chunks with a simple stand-in (hashed word vectors), and checks whether the top retrieved chunk contains the sentence that answers each question, and how much of that chunk is the answer.

Python Which chunk size retrieves best on these questions?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table carefully. At 10 words every chunk is a single sentence, so each of our one-sentence questions finds its answer (20 of 20) and the top chunk is only the answer (precision 1.00). That is a result of this test, not a rule: real questions often need two or three sentences, which a 10-word chunk cannot hold. As chunks grow, two things degrade: hit@1 falls (from 20/20 to 15, 13, 14, 14 and 13 of 20 for 20 to 320 words, because the look-alike sentences of several services are blended into one vector), and precision collapses to 0.02 at 320 words (the answer is a needle in a 280-word chunk, and the LLM must pay for all of it). The best size sits where your questions are answered whole without dragging in unrelated text, and it differs from corpus to corpus. Your job is to measure it on your own questions with the evaluation set and retrieval metrics, changing one thing at a time.

Beyond plain chunks: small to big

Two chunk sizes can give the best of both worlds. Parent-child (also called small-to-big) retrieval embeds small child chunks for precise matching, but returns the bigger parent chunk (or the surrounding paragraphs) to the LLM for context. A related trick is sentence-window retrieval: match on one sentence, return it with its two neighbors on each side. Both are covered in advanced RAG patterns.

Practical guidance

  • Start with recursive splitting, 300 to 500 tokens, 10 to 15% overlap.
  • Keep chunks self-contained: prepend the document title and section heading so a chunk like "It expires after 1 hour" still says what expires.
  • Match chunk size to the question type: lookup questions want small chunks, explanation questions want larger ones.
  • Do not split code, tables or lists in the middle (structure-aware chunking).
  • Store the chunk's position (chunk_index, start, end) and parent id as metadata, so you can fetch neighbors and cite exactly.
  • Re-evaluate whenever you change the embedding model: the best chunk size can move.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does chunk overlap solve?

    Overlap repeats the tail of one chunk at the start of the next so boundary-spanning sentences survive.
  2. 2. What is a downside of very large chunks?

    Large chunks dilute the embedding and put more irrelevant text into the prompt.
  3. 3. How does a recursive splitter work?

    It tries paragraphs, then sentences, then words, respecting structure as far as the budget allows.
  4. 4. Why prepend the title or section heading to a chunk?

    Chunks often lose their subject; a heading restores the context the embedding needs.
  5. 5. How should you pick the final chunk size?

    The best size depends on content and questions, so measure retrieval metrics on your own data.

Summary

  • Chunking balances precision (small) against context (large); both extremes hurt.
  • Fixed-size splitting is simple but cuts sentences; overlap and sentence-aware or recursive splitting fix most of that.
  • Measure chunk size in tokens, keep chunks self-contained, and store position and parent metadata.
  • Choose sizes by testing on your own questions; small-to-big retrieval can combine precision with context.

Next: Structure-Aware and Semantic Chunking