Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 63 · Cleaning and Tokenization

Building a Reusable Preprocessing Pipeline

Combine cleaning, tokenization and filtering into one reusable NLP preprocessing pipeline with composable functions, a saved configuration, a fingerprint and tests.

You now know the individual steps of text preprocessing: cleaning, tokenizing, splitting sentences, removing stop words, stemming. The danger comes when they live as scattered snippets in a notebook. Six months later nobody remembers whether the training data was lowercased, and the model in production sees text that was cleaned slightly differently. Accuracy drops, and the cause is invisible.

A preprocessing pipeline puts every step in one place so that training, evaluation and production all run the same code. In this lesson you will build one from small composable functions, drive it from a saved configuration, give each configuration a fingerprint for reproducibility, and write tests that catch regressions. The pieces come from cleaning and normalizing text and tokenization.

Why a pipeline

One pipeline, three places

Training Prepares the data the model learns from.
Evaluation Must match training exactly.
Production Sees real, messy user text.

Any difference between them is a silent bug.

1 / 2

The mismatch between training and production preprocessing is sometimes called train/serve skew. It is one of the most common reasons a model that scored well in testing disappoints in real use, and a pipeline object is the cheapest cure.

Composable steps

The first rule is that every step is a small function from one value to the next. Cleaning steps take a string and return a string; the tokenizer takes a string and returns a list. If every step has a clear input and output, a pipeline is just running them in order.

Python Small steps joined with compose

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last loop is a debugging habit worth keeping: run the steps one at a time and print after each, so you can see exactly where text changes. Note two details. The order matters: HTML must be removed before whitespace is collapsed, because removing a tag leaves extra spaces behind. And html.unescape turns   into a non-breaking space, which NFKC normalization then converts into a plain space, so the steps cooperate.

The tokenizer pattern uses [^\W_], which means "a letter or digit in any language". A pattern like [a-z0-9]+ would silently cut café into caf and drop every non-English letter. Bugs like this are exactly what tests, later in this lesson, are for.

Driving the pipeline from configuration

Hard-coded choices make experiments painful: to try a stop list you edit code, and you can no longer say which version produced which model. Better to put the choices in a small, frozen configuration object and build the pipeline from it. A frozen dataclass cannot be changed after creation, so a config cannot drift while a job runs.

Python A configuration object and a pipeline builder

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same input yields three different token lists, and each is fully described by a short piece of data. asdict plus json.dumps turns the configuration into text you can store next to the trained model, commit to version control or print in a log. Splitting what to do (config) from how to do it (the builder) is the central idea of this lesson.

Never let a config silently fall back to different defaults over time. If you add a new option later, give it a default that reproduces the old behavior so that saved configs still mean the same thing.

Reproducibility: fingerprints

How do you know two runs used the same settings? Hash the canonical JSON form of the config. Sorting keys makes the JSON identical for identical settings, so the hash identifies the configuration. Store the short fingerprint with every artifact: the vectorizer, the trained model, the evaluation report. If the fingerprint at serving time differs from the one stored with the model, refuse to start.

Python Fingerprints and round trips

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fingerprint of a and b is identical, while flipping one flag gives a completely different value, and a save-and-load round trip preserves it. The conversion from a JSON list back to a tuple in load is easy to forget: a list and a tuple of the same words would otherwise compare differently and hash differently.

Hashing a config does not make tokenization deterministic by itself, but it covers the most common failure: using different settings in different places. For full reproducibility also record library versions (such as your NLTK or scikit-learn release), because a library upgrade can change a tokenizer or a stop list.

Testing the pipeline

Preprocessing code is easy to break and hard to notice. A model may happily train on garbled tokens. Tests are cheap insurance. Write them in three kinds:

  • Example tests: known input, known output, including ugly cases (accents, emoji, HTML, empty text).
  • Property tests: facts that must hold for any input, for instance that cleaning twice gives the same result as cleaning once (idempotence), or that the tokens never contain whitespace.
  • Regression tests: whenever you find a bug, add the exact input that triggered it.
Python A tiny test runner for the pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The buggy tokenizer fails exactly on the accent and apostrophe cases, which is the kind of bug that quietly degrades a model for months. In a real project these checks live in a test file run by pytest on every change; the structure is the same. Keep the example list small and meaningful, and grow it from real bugs.

Habits that keep a pipeline healthy

  • Fit on training data only. Anything that learns from data, such as a frequency-based stop list or a vocabulary, must be computed on the training set and then reused unchanged. See text classification workflow for why leaking test data inflates scores.
  • Return tokens, keep the raw text. Store the original text so you can re-run a new config without re-collecting data.
  • Keep the pipeline small. Each step is a decision to defend. Start with cleaning and tokenizing, and add stop words or stemming only when a measurement says they help.
  • Log the fingerprint with every experiment and every deployed model.
  • Ship the same code. Import the pipeline function from one module in training scripts and in the service, and never copy and edit it.

scikit-learn can wrap such a function into a Pipeline so that preprocessing and the model travel together as one object, covered in scikit-learn pipelines for text.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What is train/serve skew in text preprocessing?

    If production text is cleaned differently from training text, the model sees inputs it was never trained on, and accuracy drops without an error message.
  2. 2. Why is the configuration made a frozen dataclass?

    An immutable config means the object that built the pipeline is the same one you saved and fingerprinted.
  3. 3. Why does the fingerprint use json.dumps with sort_keys=True?

    Without a fixed key order, equal configs could serialize differently and get different hashes.
  4. 4. A tokenizer using the pattern [a-z0-9]+ turns "café" into "caf". Which kind of test would have caught this?

    A single example with an accent shows the problem at once. After fixing, it stays as a regression test.
  5. 5. Which statistic must be computed only on the training data when it feeds the pipeline?

    Anything learned from the data would otherwise leak information from the test set.

Summary

  • A pipeline runs the same preprocessing in training, evaluation and production, which prevents train/serve skew.
  • Write small single-purpose steps and join them with a compose helper; order matters.
  • Put choices in a frozen config, build the pipeline from it and save it as JSON.
  • Fingerprint the canonical JSON and store the hash with every model and report.
  • Test with examples, properties such as idempotence, and a regression case for every bug.

Next, start counting words in Word Frequencies and Zipf's Law.