Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 63 Β· Practice and Production

NLP in Production

Ship an NLP model safely: measure latency and model size, version your artifacts, detect data drift, test text pipelines and monitor a live system.

A model that scores well in a notebook is only the beginning. In production it has to answer in milliseconds, fit in memory, survive weird input, and keep working months later while the world changes under it. Language changes fastest of all: new slang, new product names, new spam tricks, a new customer base writing in a different style.

This lesson covers the engineering around an NLP model: how to measure speed and size, how to version what you ship, how to notice drift, how to test a text pipeline, and what to monitor. The examples use small scikit-learn models so you can run everything here, but every idea applies equally to a transformer behind an API.

The production checklist

Five things that break in production

Latency Users wait for every prediction.
Size Big models cost memory and startup time.
Versions Know exactly which model answered.
Drift Today's text is not training text.

Testing and monitoring watch over all four.

1 / 2

Latency and throughput

Latency is how long one request takes. Throughput is how many requests you finish per second. They are related but not the same: batching many texts into one call usually raises throughput a lot while each individual text may wait slightly longer.

Where does the time go? For a classic model, mostly in preprocessing and vectorizing, not in the final prediction. For a neural model, mostly in the forward pass, and it grows with the length of the text. Always measure; never guess. Use time.perf_counter(), repeat the measurement, and take the best of a few runs, because the first call and background noise distort single timings.

The example below trains a tiny sentiment model, then predicts 200 texts one at a time and in one batch. Timings differ from machine to machine, so the code prints comparisons instead of raw seconds. It also shows how much a bigger feature space costs in size.

Python Latency, batching and model size

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Batching wins because Python and NumPy pay a fixed cost per call, and one big call spreads it over many texts. The size lines show the other trade-off: on this tiny data, adding bigrams and trigrams grows the vocabulary from 28 to 70 terms and the pickled model from about 2.0 KB to 3.4 KB. On real data with millions of n-grams the same effect turns a 2 MB model into a 2 GB one.

Typical ways to cut latency and size, roughly in the order you should try them:

  • Cache results for repeated inputs. Real traffic repeats itself a lot.
  • Batch requests, or process them asynchronously when users do not need an instant answer.
  • Truncate long inputs to the length that actually matters for the task.
  • Shrink the model. Use min_df, max_features or feature hashing for classic models. For neural models, use a smaller architecture, quantization (storing weights in 8 bits instead of 32) or distillation (training a small model to imitate a big one).
  • Use the cheap model first. A fast classifier can handle the easy 90 percent, and only unsure cases go to an expensive model.

Report latency as percentiles, not averages. The p50 says what a typical user sees; the p95 and p99 show the slow tail that actually causes complaints.

Versioning: know what answered

When a prediction looks wrong, the first question is always "which model produced it?". To answer that, treat the model like a release. Store, next to the weights:

  • the library versions (scikit-learn, transformers, numpy), because pickles and weights can break across versions,
  • a fingerprint of the preprocessing and vocabulary,
  • the training data identifier and date,
  • the evaluation numbers measured at release time.

Never rely on "the latest file in the folder". A model artifact should be immutable, and the service should log the version string with every prediction.

A short hash makes a good fingerprint, because any change to the vocabulary changes it. Here we also run the same artifact checks a deployment script would run before accepting a model.

Python A model card and a release check

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The card is plain JSON you can store beside the model and load in the service. The release check uses a golden set: a few examples whose answers you are sure about. A swapped label file produced a model that fails the golden set immediately, and the check blocks the release. Golden sets are cheap and catch an astonishing number of deployment mistakes.

Drift: when the world moves

A model learns the language of its training data. Production text slowly stops looking like that. Two kinds of drift matter:

  • Data drift (covariate shift): the inputs change. New words appear, texts get longer, a new language shows up.
  • Concept drift: the meaning of the labels changes. What counted as spam last year is different from this year's spam, or "sick" starts to mean "great".

You usually have no labels in production, so you watch proxies that need none:

  • the out-of-vocabulary (OOV) rate: the share of incoming tokens the model has never seen,
  • the length distribution of texts,
  • the distribution of predicted classes (a sudden jump in "spam" is a signal),
  • the average model confidence.

When a proxy crosses a threshold, you sample some texts, label them by hand, and measure real accuracy. That confirms whether to retrain.

flowchart LR
    T["Live traffic"] --> M["Model"]
    M --> L["Log: version, input stats, prediction, confidence"]
    L --> D["Drift monitor"]
    D -->|"threshold crossed"| S["Sample and label texts"]
    S --> R["Retrain and release a new version"]
    R --> M

Monitoring closes the loop: logs feed a drift check, and drift triggers labeling and retraining.

The next example compares two weeks of traffic with the training vocabulary. Week 2 contains new terms the model has never seen.

Python Detecting vocabulary drift

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Week 1 looks like training data: only 6 percent of its tokens are new (the word high). Week 2 talks about an app that keeps crashing after an update, vocabulary the model has never met, so the rate jumps to 67 percent and raises an alert. The list of new words is the most useful part: it tells you what the data is about now. Pick the threshold from history: measure the normal OOV rate for a few weeks and alert on a clear departure from it.

Subword models such as BPE never have a truly unknown word, so OOV rate loses meaning for them. For those, track the average number of tokens per word instead: it rises when the text becomes unfamiliar.

Testing text pipelines

Text code has its own classic bugs: empty strings, None, emoji, accents, very long lines, mixed languages, and inputs that look different but are the same after normalization. Tests for an NLP service come in layers:

  1. Unit tests for each preprocessing step (cleaning, tokenizing) with exact expected outputs.
  2. Property tests: things that must always hold, such as "cleaning twice gives the same result as cleaning once" (idempotence) or "the output never contains uppercase letters".
  3. Golden-set tests for the model, as you saw above, with a minimum accuracy.
  4. Behavioral tests: small perturbations that must not change the answer (extra whitespace, a different casing) or must change it in a known way (adding "not" flips sentiment).

The code below tests a tiny cleaning function with all four ideas, using plain assert statements and a mini test runner so you can see each result.

Python Testing a text pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every test passes, and two of them deserve attention. test_unicode_forms_match checks the classic trap from Levels of Language: the same visible word can be stored in two different ways, and unicodedata.normalize makes them equal. test_idempotent guards against a bug that is easy to introduce: a cleaning step whose output the next run changes again, which makes training and serving disagree if one of them cleans twice. In a real project you would put these functions in a tests/ folder and let pytest run them on every commit; here we call them by hand.

The most important test of all is training-serving consistency: the exact same preprocessing code must run in training and in the service. Keep it in one importable function or, better, inside the saved pipeline object, and never copy it.

What to monitor

Once the service is live, log enough to debug without logging more than you should. For every request, record the model version, text length, latency, predicted label and confidence. Dashboards then show:

  • request volume and error rate,
  • latency percentiles,
  • prediction distribution over time,
  • OOV rate or tokens per word,
  • confidence histogram,
  • user feedback signals, such as corrections and "this was not helpful" clicks, which are free labels.

Be careful with privacy. Texts often contain names, emails, addresses and medical or financial details. Log hashes or statistics rather than raw text when you can, set a retention period, and follow your legal obligations. See Evaluation, Bias and Responsible NLP for the broader picture.

A deployment sketch (read-only)

A common minimal setup wraps the saved pipeline in a small web service. This code does not run in the browser; it shows the shape of the service:

Python
# service.py -- read-only sketch, run with: uvicorn service:app
import time
import joblib
from fastapi import FastAPI
from pydantic import BaseModel, Field

MODEL_VERSION = "1.0.0"
model = joblib.load(f"models/sentiment-{MODEL_VERSION}.joblib")  # loaded once at startup
app = FastAPI()

class Request(BaseModel):
    text: str = Field(max_length=5000)        # reject absurdly long input early

@app.post("/predict")
def predict(req: Request):
    start = time.perf_counter()
    proba = model.predict_proba([req.text])[0]
    label = int(proba.argmax())
    return {
        "label": label,
        "confidence": float(proba[label]),
        "model_version": MODEL_VERSION,
        "latency_ms": round((time.perf_counter() - start) * 1000, 2),
    }

The key habits are visible even in this short file: load the model once at startup instead of per request, limit input size, return the model version, and measure latency. Around it you would add health checks, request logging, rate limits, a container image with pinned dependencies, and a rollout that sends a small share of traffic to a new version first (a canary release) before switching everything.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. Why does batching usually increase throughput?

    Each call has fixed costs. Processing many texts in one call shares those costs, so more texts are finished per second.
  2. 2. Which signal can you watch for drift when you have no labels in production?

    Labels are rarely available live, so you monitor proxies computed from the inputs and outputs alone, then label a sample when they move.
  3. 3. What is the purpose of a golden set in a release check?

    A small set of known-answer examples with a minimum accuracy quickly exposes swapped labels, wrong vocabularies and other deployment mistakes.
  4. 4. What does it mean that a cleaning function is idempotent?

    Idempotence means clean(clean(x)) equals clean(x). Without it, text that is cleaned twice (once in training and again in serving, say) can differ.
  5. 5. Why should the same preprocessing code run in training and in serving?

    Any difference in cleaning or tokenization between training and serving silently lowers accuracy. Keeping it in one function or one pipeline object prevents that.

Summary

  • Measure latency (as percentiles) and throughput; batching, caching, truncation and smaller models are the main levers.
  • Model size grows with the feature space; quantization, distillation and feature hashing shrink it.
  • Version every artifact with library versions, a vocabulary fingerprint, training data info and release metrics, and log the version with each prediction.
  • Drift is inevitable: monitor OOV rate (or tokens per word), length, predicted classes and confidence, then label a sample to confirm.
  • Test text pipelines with unit, property, golden-set and behavioral tests, and keep preprocessing identical in training and serving.
  • Log what you need for debugging while protecting privacy.

Now put everything together in a complete application, starting with Project: Spam Filter.