Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 36 Β· Chunking and Ingestion

Loading and Cleaning Documents

Turn HTML, PDF text and mixed files into clean documents for RAG: extract text, strip boilerplate, fix hyphenation and Unicode, and add quality checks before chunking.

The embedding model and the language model can only work with the text you give them. If a scraped page contributes "Home | Pricing | Login | Accept cookies" to every chunk, or a PDF breaks words like "recom-\nmendation" in half, retrieval quality drops long before any clever indexing matters. The first stage of every RAG pipeline is therefore unglamorous and decisive: load the documents and clean them.

This chapter walks through the ingestion side: cleaning (this lesson), chunking, structure-aware splitting and a complete ingestion pipeline. All code runs with the Python standard library, so you can try it here.

What a loader must produce

A loader turns a source (file, URL, database row) into one or more documents, each with:

  • text: clean, plain text (or Markdown that preserves headings and lists),
  • metadata: where it came from (source, title, url), when (updated_at), who may see it (access), and anything you may want to filter on (language, product, version),
  • a stable id, so that re-ingesting the same source updates instead of duplicating (updates).

Metadata is easiest to capture at load time. Once text is split into chunks, the context is gone.

Source types and their traps

Source Typical problems
HTML pages Navigation, footers, cookie banners, scripts, ads; the useful content is a small part of the markup.
PDF Text extracted in odd reading order, columns interleaved, hyphenated line breaks, repeated headers and footers, tables flattened, scanned pages with no text at all (need OCR).
Word / PowerPoint Tables, comments, tracked changes, speaker notes.
Markdown / wiki Front matter, macros, relative links; generally the cleanest source.
JSON / CSV / database rows You decide what a "document" is: one row? one record with its related rows?
Chat logs, tickets, email Quoted replies repeated many times, signatures, threads.

Prefer a source that already has structure (an API, a Markdown repository, a database) over scraping rendered pages, and prefer the original document over a PDF export of it.

Extracting text from HTML

The standard library's HTMLParser is enough to see the idea: walk the tags, ignore the ones that never contain article content, keep headings as markers (they help structure-aware chunking) and add blank lines between blocks.

Python HTML to clean text, keeping headings

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The head (title and scripts), navigation, the sidebar and the footer are gone; the headings survive as # markers and the list becomes bullet lines. Real sites need a smarter "main content" extractor (libraries exist for this), but the principle is the same: keep the article, drop the chrome.

Cleaning text extracted from PDFs

Text from PDFs is the messiest. Typical artifacts:

  • Hyphenation: words broken at line ends (recom- newline mendation).
  • Hard line wraps: every visual line ends with a newline, splitting sentences.
  • Repeated headers and footers: "ACME Handbook 2025 - Confidential" and page numbers on every page, which then pollute every chunk.
  • Ligatures and odd characters: fi, non-breaking spaces, smart quotes, zero-width characters.

The following pipeline fixes each one. Headers and footers are found by a robust trick: a line that appears on most pages is boilerplate.

Python A cleaning pipeline for extracted PDF pages

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output has no header, no page numbers, joined words (calendar, manager, first) and sentences that flow. One flaw is visible: the heading "Leave policy" got glued to the sentence after it, because a line without final punctuation looks like a wrapped line. Real extractors use font sizes to detect headings. Heuristics like these are not perfect (a genuine hyphenated word such as "state-of-the-art" at a line end would be glued), so sample your cleaned output by eye and keep the rules conservative.

Scanned PDFs contain images rather than text; they need OCR (optical character recognition), and OCR output needs the same cleaning plus spell-check awareness. Tables deserve special care: flattened table text loses its meaning, so extract tables separately (as Markdown or JSON rows) and index them as their own chunks.

Loaders and quality checks

Wrap all sources behind one interface so the rest of the pipeline does not care. Then add quality gates: documents that fail them are logged, not silently indexed.

Python A loader interface with quality checks

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every document carries an id derived from its source, a title and a type; failures are reported with a reason. In a real pipeline you would send the skipped documents to a log or a review queue, then fix the extractor or the source.

A cleaning checklist

  • Extract the main content; drop navigation, banners and repeated page furniture.
  • Normalise Unicode and whitespace; repair hyphenation and wrapped lines.
  • Preserve structure that carries meaning (headings, lists, tables, code blocks) as Markdown.
  • Keep provenance: source, URL, title, version, last-updated date, permissions.
  • Remove or mask what must never be indexed (security): secrets, personal data, internal-only pages.
  • Run quality gates and read a random sample of cleaned text before every large ingestion.
flowchart LR
    S["Source (HTML, PDF, MD, DB)"] --> E["Extract text and structure"]
    E --> C["Clean: boilerplate, Unicode, wraps"]
    C --> M["Attach metadata and id"]
    M --> Q{"Quality gate"}
    Q -->|pass| N["To chunking"]
    Q -->|fail| L["Log and review"]

The document stage of an ingestion pipeline.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. Why is it risky to leave 'Home | Pricing | Login' text in documents?

    Boilerplate repeated across pages pollutes embeddings and retrieval results, and wastes prompt tokens.
  2. 2. How can you automatically detect repeated page headers and footers in extracted PDF pages?

    Counting how many pages contain each line finds furniture reliably without hand-written rules.
  3. 3. Why should metadata be captured when loading documents?

    Source, title, version, and permissions enable filtering, citations and updates later.
  4. 4. A scanned PDF yields no text. What is needed?

    Scanned pages are images; OCR must produce text before any other step.
  5. 5. What is the benefit of a stable document id derived from the source?

    Stable ids make re-ingestion idempotent and enable updates and deletes.

Summary

  • Clean, well-sourced text is the foundation of retrieval quality; do this stage deliberately.
  • Extract main content from HTML, repair PDF artifacts (headers, hyphenation, wraps, ligatures) and normalise Unicode.
  • Preserve useful structure and capture metadata and stable ids at load time.
  • Add quality gates, log failures, and always read a sample of the cleaned output.

Next: Chunking Strategies