Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Training and Scaling

Data Pipelines for Pretraining

See how pretraining corpora are built: crawling, cleaning, quality filters, exact and near-duplicate removal with MinHash, decontamination, data mixes and packing.

A model can only learn what its data contains, so the pretraining corpus is among the most important (and least published) parts of an LLM. Raw web text is noisy: boilerplate, spam, duplicates, broken encodings, and personal data. Turning trillions of raw tokens into a clean training set is an engineering pipeline of its own. This lesson walks through the stages and implements the key ideas on small examples.

The pipeline at a glance

flowchart LR
    A["Sources<br/>web crawl, books, code,<br/>papers, forums"] --> B["Extract text<br/>strip HTML, detect language"]
    B --> C["Heuristic filters<br/>length, symbols, repetition"]
    C --> D["Quality and safety<br/>classifiers, PII removal"]
    D --> E["Deduplicate<br/>exact and near-duplicate"]
    E --> F["Decontaminate<br/>remove benchmark text"]
    F --> G["Mix and tokenize<br/>weights per source, packing"]

Every stage removes data; the final corpus is usually a small fraction of the raw crawl.

Heuristic quality filters

Simple rules remove large amounts of junk cheaply: too short or too long documents, a high ratio of symbols, lines that are mostly bullet points or "read more" links, extreme word repetition, and wrong language.

Python Rule-based quality filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Thresholds like these are tuned on samples and are deliberately simple; production pipelines pair them with learned classifiers that score how "educational" or "reference-like" a page is, often trained on text that humans or a strong model rated highly.

Exact deduplication

The web repeats itself: licenses, mirrored sites, scraped copies. Training on duplicates wastes compute, encourages memorization, and can leak the same text into train and test. The cheapest step is exact deduplication: hash each (normalized) document and keep one per hash.

Python Exact dedup by hashing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Near-duplicate removal with MinHash

Many duplicates differ slightly: a changed date, an extra footer. Exact hashing misses them. Jaccard similarity between the sets of word shingles (overlapping n-grams) of two documents measures overlap, but comparing every pair of billions of documents is impossible. MinHash compresses each document into a short signature such that the fraction of matching signature entries estimates the Jaccard similarity, and locality-sensitive hashing groups likely matches without all-pairs comparison.

Python MinHash estimates Jaccard similarity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate tracks the true similarity with 128 hashes (it is a noisy estimate, so expect it to be close rather than identical). A threshold such as 0.8 decides which near-duplicates to drop. Studies of deduplication show it reduces memorization and can improve results for the same compute.

Decontamination

Public benchmark questions and answers appear all over the web. If they end up in the training set, test scores overstate real ability. Decontamination removes training documents that share long n-grams (for example 13-grams) with benchmark items. It is imperfect, because paraphrases slip through, which is why reported scores are treated cautiously (see the evaluation chapter).

Python Find overlapping n-grams with a test set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mixing sources

The final corpus is a mixture: web text, books, code, scientific papers, math, multilingual text, and conversations, each with a sampling weight. Upweighting high-quality sources, repeating them a few times, or putting the best data late in training (a "curriculum" or annealing phase) can matter as much as raw size. Code and math in the mix help reasoning even on non-code tasks.

Python Sampling from a weighted data mix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Packing and tokenization

Documents are tokenized, joined with the EOS token between them, and packed into fixed-length training sequences (for example 4,096 or 8,192 tokens) so no compute is wasted on padding. Some systems mask attention at document boundaries so a document does not attend to the unrelated text packed before it.

From crawl to corpus

Filter Rules and classifiers drop junk.
Deduplicate Exact hashes and MinHash remove copies.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why remove duplicate documents from training data?

    Repeated text is overweighted, memorized more easily and can contaminate evaluations.
  2. 2. What does MinHash let you do?

    The fraction of matching minimum hashes estimates the Jaccard similarity of two shingle sets.
  3. 3. What is benchmark decontamination?

    It prevents the model from memorizing test answers, which would inflate scores.
  4. 4. Why pack documents into fixed-length sequences?

    Packing fills each sequence with real tokens separated by EOS.

Summary

  • Pretraining data goes through extraction, filtering, deduplication, decontamination, mixing and packing.
  • Heuristic and learned filters remove low-quality text.
  • Exact hashing and MinHash remove duplicates; shared long n-grams reveal benchmark contamination.
  • Source mixture weights and data order shape what the model learns.
  • Quality and diversity often matter more than raw volume.

Next: Scaling Laws and Compute