Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 36 ยท Chunking and Ingestion

Metadata and Ingestion Pipelines

Design a reliable RAG ingestion pipeline: chunk metadata, stable ids and content hashes, exact and near-duplicate removal with MinHash, incremental sync, batching and error handling.

A demo ingests a folder of files once. A production system ingests continuously: documents change, pages are deleted, the same text appears in five places, an embedding API times out halfway through. The difference between a demo and a service is the ingestion pipeline: a repeatable, idempotent, observable process that turns sources into indexed chunks.

This lesson designs that pipeline end to end: the metadata every chunk needs, how to detect what changed so you only re-embed what you must, how to remove duplicates, and how to run the work in batches without corrupting the index. You have seen the stages in this chapter already (cleaning, chunking, structure); here they are connected.

flowchart LR
    A["Sources"] --> B["Load + clean"]
    B --> C["Chunk + attach metadata"]
    C --> D["Deduplicate"]
    D --> E["Diff against the index"]
    E --> F["Embed new/changed chunks in batches"]
    F --> G["Upsert + delete stale"]
    G --> H["Report and log"]

An ingestion pipeline: every stage can fail alone, so each must be safe to repeat.

Metadata: what to store with each chunk

The vector alone is useless to your application. Alongside it, store a record:

Field Purpose
chunk_id Stable unique id (see below).
doc_id, source, url Where it came from; needed for citations.
title, breadcrumb Human-readable context; useful in the prompt and UI.
chunk_index, parent_id Position and neighbors, for context expansion.
text The chunk itself (or a pointer to it).
content_hash Hash of the text, for change detection and deduplication.
embedding_model, embedded_at So you know which vectors to refresh after a model change (versioning).
updated_at, version, valid_until Freshness filters and rollbacks.
access (tenant, roles, groups) Permission filters (security).
language, product, doc_type Filterable attributes (filtering).

Keep metadata small and typed (strings, numbers, dates, booleans, short lists). Vector databases index these fields for filtering; storing large blobs there slows everything.

Stable ids and content hashes

A pipeline that generates random ids creates duplicates on every run. Instead derive ids deterministically:

chunk_id = hash(doc_id + chunk_index) identifies the place, and content_hash = hash(text + embedding_model) identifies the content. Together they answer: "Is this chunk new, changed, or unchanged?"

Python Incremental sync: only embed what changed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sync touches exactly two chunks (one changed, one new) and skips the rest, so re-running it over unchanged content costs nothing. Note the hash includes the embedding model name: after a model change every hash differs and everything is re-embedded once, which is correct. The delete rule is limited to managed documents, so a bug that returns an empty source list cannot wipe unrelated data (always put a sanity limit on how many deletions one run may perform).

Duplicates: exact and near

Duplicate chunks waste storage and, worse, fill the top-k with five copies of the same passage, pushing out other useful results (reranking and MMR mitigate this at query time, but prevention is cleaner). They come from copied pages, boilerplate, versions of the same policy and mirrored sites.

  • Exact duplicates: same content_hash after normalisation (lowercase, collapse whitespace). Keep one record and list all sources in its metadata.
  • Near-duplicates: same text with small edits (a date, a version number). Detect with MinHash: represent each text as a set of word shingles (overlapping word groups), compute a short signature whose agreement rate estimates the Jaccard similarity of the sets, and compare signatures instead of full texts. It is the set-similarity cousin of locality-sensitive hashing.
Python Near-duplicate detection with MinHash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two policy versions are flagged as near-duplicates even though the number changed, while the unrelated page is distinct. What you do next is a policy decision: keep the newest version and drop the old one (versions of a fact), or keep both with a version metadata field if users must be able to ask about past versions. For large corpora the pairwise comparison is avoided by banding the signatures into hash buckets (the LSH trick) so only likely pairs are compared.

Running the pipeline reliably

  • Idempotent: running it twice leaves the same state. Ids and hashes make that natural.
  • Batch: embed 64 to 512 chunks per API call, within the provider's limits. Use bounded concurrency and respect rate limits.
  • Retry with backoff on timeouts and 429s; fail the batch, not the whole run, and report what failed.
  • Order matters: embed and upsert new chunks first, delete stale ones after, so a failure never leaves a document with no chunks.
  • Dry run: print the add/update/delete counts and a sample before writing, and cap deletions (for example abort if more than 20% of the index would go).
  • Incremental triggers: webhooks or change feeds from the source beat nightly full crawls; keep a nightly reconciliation to catch missed events.
  • Observability: log counts per stage, rejected documents with reasons, embedding latency and cost, and time since each source was last synced (observability).
  • Backfill and re-embedding: when the model or chunker changes, build a new index alongside the old one, validate it on your evaluation set, then switch traffic (updates and versioning).

A pipeline skeleton

Python A complete (small) ingestion run with a report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run 1 embeds every chunk and rejects the stub; run 2 embeds nothing because every hash matches; run 3 re-embeds only the chunk that contains the edit. That behavior (cheap, repeatable, reported) is the target for any real pipeline, whether you write it yourself or use a framework. Frameworks can save time, but the design questions are the same: stable ids, content hashes, metadata, batching, retries, deletions with a safety cap and a report you can read.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why include the embedding model name in the content hash?

    Vectors from different models are incompatible, so a new model must re-embed all chunks.
  2. 2. What makes an ingestion run idempotent?

    With deterministic ids and hashes, re-running changes nothing unless the source changed.
  3. 3. What does MinHash estimate?

    The fraction of matching signature positions approximates the Jaccard similarity.
  4. 4. Why upsert new chunks before deleting stale ones?

    Adding first keeps the old content available until the new content is safely in place.
  5. 5. Why cap the number of deletions in one run?

    A safety cap turns a catastrophic mistake into a refused run with a clear report.

Summary

  • Every chunk needs a stable id, a content hash and metadata for citations, filtering, permissions and versioning.
  • Compare hashes against the index to embed only what is new or changed and to find stale chunks.
  • Remove exact duplicates by hash and near-duplicates with MinHash; decide a policy for versions.
  • Run in batches with retries, upsert before delete, cap deletions, and produce a report for every run.
  • Re-embedding after a model or chunker change builds a new index that you validate before switching.

Next: Retrieval Basics, Top-k and Score Thresholds