Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 36 ยท Chunking and Ingestion

Structure-Aware and Semantic Chunking

Split by headings, code blocks and tables, add heading breadcrumbs, detect topic shifts with embeddings, and retrieve small chunks while returning their larger parents.

Fixed-size and recursive splitters know nothing about meaning: they see characters and punctuation. Yet most documents come with a map of their own meaning. A manual has chapters, sections and tables; an API reference has one entry per function; a transcript changes topic every few minutes. Splitting along those natural seams gives chunks that are about one thing, which is exactly what makes embeddings sharp.

This lesson covers three ideas that build on each other: structure-aware splitting (use headings and blocks), semantic splitting (find topic shifts with embeddings), and parent-child retrieval (match small, return big).

Structure-aware chunking: split on headings

Markdown, HTML and Word documents carry heading levels. A structure-aware splitter starts a new chunk at each heading and attaches the heading path, called a breadcrumb, to the chunk. The breadcrumb gives the chunk the context it would otherwise lose ("Billing > Refunds > Annual plans").

Rules worth following:

  • Never split inside a code block or table; keep it whole (if it is too large, split between rows or statements and repeat the table header).
  • Keep a list with the sentence that introduces it.
  • If a section is larger than your budget, split it further with the recursive splitter, but keep the same breadcrumb on every piece.
  • Put the breadcrumb in the embedded text (and store it as metadata too). It improves matching and helps the LLM understand the chunk.
Python A Markdown splitter with heading breadcrumbs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice two things in the output: the # list projects comment inside the code fence did not start a new section, and every chunk starts with its breadcrumb. A chunk that says only "Tokens expire after 90 days" now arrives labelled "Orbit Guide > API > Authentication".

Other structures follow the same logic. HTML: split on h1 to h4, keep table and pre intact. Source code: split by function or class (a language-aware parser beats text splitting). Slides: one slide per chunk with the slide title. Chat and tickets: one message or one thread per chunk, with the author and date. FAQ pages: one question and answer per chunk.

Semantic chunking: find where the topic changes

Not every text has headings (transcripts, long emails, plain articles). Semantic chunking uses embeddings to find topic boundaries:

  1. Split the text into sentences.
  2. Embed each sentence (often with a small context window of neighbors).
  3. Compute the similarity between each sentence and the next.
  4. Where similarity drops sharply, a topic probably changes: cut there.
flowchart LR
    S["Sentences"] --> E["Embed each"]
    E --> D["Similarity of neighbors"]
    D --> T["Find big drops (below a percentile)"]
    T --> C["Cut chunks at the drops, respect min/max size"]

Breakpoints are the places where adjacent sentences are least alike.

The stand-in below gives a handful of topic words a direction (a tiny hand-made lexicon) instead of calling a real sentence embedding model; everything after the embedding step is exactly what a real system does.

Python Semantic chunking by similarity drops

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Neighbors inside a topic score 0.65 to 0.95, while the two topic switches score -0.13 and -0.05, so the cuts fall after sentences 3 and 6 and the three topics (coffee, app sync, bread) end up in separate chunks. On real text expect a few wrong cuts, so semantic chunkers add guard rails: a minimum and a maximum chunk size, a sliding window to smooth noisy sentence similarities, and a percentile or standard-deviation threshold instead of a fixed number.

Semantic chunking costs an embedding call per sentence at ingestion time, and its gain over a good recursive splitter with headings is modest and corpus-dependent. Use it where structure is missing; measure before you commit (evaluation).

Parent-child (small-to-big) chunks

Retrieval prefers small, focused chunks; generation prefers rich context. Resolve the conflict by storing both:

  • Child chunks (one to three sentences) are embedded and searched.
  • Parent chunks (the whole section or a few paragraphs) are stored by id and are what the LLM receives.

Every child carries parent_id. At query time: search children, collect their parent ids, de-duplicate, and fetch the parents.

Python Match small, return big

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The match was made on one precise sentence; what goes to the LLM is the full section, including the neighboring sentence about monthly plans that the question did not mention but that prevents a wrong answer. Variants: sentence-window retrieval returns the matched sentence plus N neighbors; auto-merging returns the parent when several of its children match. Both appear in advanced RAG patterns.

Choosing a strategy

Content Start with
Manuals, docs, wikis, Markdown Heading-aware split + breadcrumbs, recursive split for long sections
API references, code One chunk per function/endpoint/class
FAQs, tickets One Q and A or one thread per chunk
Transcripts, long prose without headings Recursive splitting; try semantic chunking and compare
Tables and spreadsheets Row groups with the header repeated, or a text summary per table
Anything where answers need context Parent-child or sentence-window retrieval

Whatever you choose, store breadcrumb, chunk_index, parent_id and source as metadata, and test the choice on your evaluation questions.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a heading breadcrumb in a chunk?

    The breadcrumb restores the context a section-level chunk would otherwise lose and can be embedded with the text.
  2. 2. Why must a structure-aware splitter track code fences?

    Treating comments in code as headings would cut code blocks apart.
  3. 3. In semantic chunking, where do you place chunk boundaries?

    A drop in neighbor similarity signals a topic change.
  4. 4. What is the idea of parent-child retrieval?

    Small chunks give precise matching; the bigger parent gives the model enough context.
  5. 5. Which content most clearly suits 'one chunk per function'?

    Code has natural units (functions, classes, endpoints); splitting by them keeps each chunk coherent.

Summary

  • Use a document's own structure: split at headings, keep code blocks and tables whole, and carry a breadcrumb in every chunk.
  • Semantic chunking finds topic shifts from embedding similarity drops; add size guard rails and compare against simpler methods.
  • Parent-child and sentence-window retrieval match precisely and return context.
  • Choose by content type, store structural metadata, and test the choice.

Next: Metadata and Ingestion Pipelines