Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 38 ยท Retrieval and Context Engineering

Chunking Documents for Prompts

Learn how to split documents into chunks for retrieval: fixed size, sentence and heading-based splitting, overlap, metadata and how to choose sizes.

You cannot put a 200-page manual into every prompt, and you should not retrieve it whole either. Before retrieval can work, documents are cut into chunks: pieces small enough to fit several into a prompt, and large enough to make sense on their own.

Chunking looks like plumbing but it decides answer quality. A chunk that cuts a sentence in half loses its meaning; a chunk that mixes five topics matches every question badly. This lesson shows the common strategies as real Python you can run, and how to choose between them.

What makes a good chunk

A good chunk is:

  • Self-contained: a reader who sees only this chunk can understand it. "It expires after 30 days" is useless without knowing what "it" is.
  • Focused: about one topic, so its relevance score is meaningful.
  • Sized for your window: with the budget in mind, you want to fit roughly three to eight chunks.
  • Traceable: it carries metadata (source file, title, section, date) so answers can cite it.

Typical sizes are 100 to 500 words. Smaller chunks are precise but lose context; larger chunks keep context but dilute relevance and cost more tokens.

Strategy 1: fixed size with overlap

Cut every N words and let neighbouring chunks share a few words, so an idea split at a boundary appears whole in at least one chunk.

Python Fixed-size chunks with overlap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fixed size is simple and predictable, but it ignores meaning. Notice how chunks start and end mid-sentence. Use it as a baseline or for text with no structure.

Strategy 2: sentence-aware packing

Split into sentences, then pack whole sentences until a size limit is reached. No sentence is ever cut.

Python Pack whole sentences into chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A single sentence longer than the limit still becomes its own oversized chunk here; real splitters add a fallback that cuts it.

Strategy 3: structure-aware splitting

Documents usually have structure: headings, lists, paragraphs, code blocks. Splitting on that structure keeps related text together. For Markdown, split at headings and prepend the heading path to each chunk so it stays self-contained.

Python Split Markdown by headings and keep the path

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The chunk "You can cancel at any time." now begins with Billing > Cancelling, so a retriever knows it is about billing, and the model can cite the section.

Metadata and parent context

Store more than text. Useful fields: source, section, updated_at, language, access_level. Metadata supports filtering ("only chunks updated this year", "only what this user may see") and citations. A popular refinement is small-to-big retrieval: search over small chunks for precision, then send the larger parent section to the model for context.

Choosing and testing a strategy

Content Good starting point
Prose, articles Sentence packing, 150-300 words, small overlap
Markdown docs, wikis Heading-based, heading path in each chunk
FAQs One question plus its answer per chunk
Code One function or class per chunk, with file path
Tables, CSV One row or row group, with the header repeated

Do not guess; test. Build a small set of real questions with the chunk that should answer each, then measure how often retrieval finds it for each strategy. The evaluation chapter shows how.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use overlap between fixed-size chunks?

    Shared words at the edges keep sentences and ideas that straddle a boundary intact in at least one chunk.
  2. 2. What does prepending the heading path to a chunk achieve?

    A line like 'It expires in 30 days' only makes sense with its section title, which also helps retrieval and citations.
  3. 3. A chunk is too large. What is the usual consequence?

    Large chunks mix topics, so their scores are less precise, and they consume more of the budget.
  4. 4. How should you choose a chunking strategy?

    Measure how often the right chunk is retrieved for a set of real questions; content type changes what works.

Summary

  • Chunks must be self-contained, focused, budget-sized and traceable.
  • Fixed size is the simple baseline; sentence packing avoids cut sentences; structure-aware splitting keeps topics together.
  • Add heading paths and metadata for citations and filtering.
  • Choose by content type, then verify with real questions.

Next: Ranking and Assembling Context