Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 63 ยท Text Classification

scikit-learn Pipelines for Text

Chain a vectorizer and a classifier with Pipeline, combine several columns with ColumnTransformer, and tune everything with GridSearchCV and cross-validation.

A text classifier is never just a model. There is a vectorizer that turns strings into numbers, maybe a feature selector, then the classifier. If you run these steps by hand, you must remember to fit each one on the training data only, apply the same transformations to new text, and repeat everything correctly inside every cross-validation fold. One slip, and your accuracy number is quietly wrong.

A scikit-learn Pipeline solves this by packaging the steps into a single object with one fit and one predict. In this lesson you will build pipelines, see the data leak they prevent, mix text and non-text columns with ColumnTransformer, and tune the whole thing with GridSearchCV.

Why pipelines?

One object Vectorizer and model travel together.
No leaks Each step is fitted on training data only.
Tunable Search settings of every step at once.
Deployable Save one file, predict from raw text.
1 / 2

Your first pipeline

A Pipeline takes a list of (name, step) pairs. Every step except the last must be a transformer (it has fit and transform); the last is the estimator (it has predict). Calling pipe.fit(texts, labels) fits the vectorizer, transforms the texts, then fits the classifier on the result. Calling pipe.predict(new_texts) runs the same chain with the already-learned vocabulary.

Python A TF-IDF and logistic regression pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pipeline scores 0.900 on the 30 held-out documents and classifies a raw sentence directly, with no manual vectorizing. named_steps gives access to any fitted step, here the vocabulary of 43 words.

The last four lines run cross-validation: the training data is split into five folds, and each fold takes a turn as the validation set while the model trains on the other four. The five scores range from 0.833 to 0.944, with a mean of 0.922 and a standard deviation of 0.044. One split gives you one number; cross-validation gives a number and an idea of how much it wobbles. Because cross_val_score receives the whole pipeline, the vectorizer is re-fitted inside every fold, which is exactly right.

StratifiedKFold keeps the class proportions the same in every fold, which matters for imbalanced data. For classifiers, passing an integer such as cv=5 already stratifies by default; we create the object explicitly to fix the shuffling seed.

The leak a pipeline prevents

Leakage happens when information from the validation data influences training. The classic text example is feature selection. Suppose you keep the 30 words that best predict the label, choosing them using all the data, and then cross-validate. The selected words were picked by looking at the validation labels, so the score is inflated.

To make the effect obvious, the next example uses documents of random words with random labels. There is nothing to learn, so an honest score should be near 0.5.

Python Leaky selection versus a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky version reports 0.817 on pure noise, which would look like a working model. The pipeline version reports 0.517, a coin flip, which is the truth. Nothing about the model changed; only where the feature selection ran. With plain TF-IDF the leak from fitting the vectorizer early is usually small, but with any step that looks at labels it can be huge. Make a habit of putting every learned step inside the pipeline.

Several columns with ColumnTransformer

Real data rarely has just one text field. A support ticket has a title, a body, a category and a length. Each column needs its own preparation: TF-IDF for each text field (they should have separate vocabularies), one-hot encoding for categories, scaling for numbers. ColumnTransformer applies a different transformer to each column and concatenates the results into one feature matrix.

Notice two details. A text vectorizer is given a column name as a string ("title"), because it expects a one-dimensional sequence of documents. Other transformers receive a list (["source"]), because they expect a 2D table.

Tuning with GridSearchCV

Every setting of every step is a hyperparameter: the n-gram range, sublinear_tf, C. GridSearchCV tries all combinations, scores each with cross-validation, and refits the best one on all the training data. Pipeline parameters are addressed with double underscores: prep__body__ngram_range means "the ngram_range of the transformer named body inside the step named prep".

Python ColumnTransformer and GridSearchCV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The preparation step produces 77 features: two vocabularies, three one-hot columns for source, and one scaled number. Twelve combinations are tried (2 x 2 x 3), each with four folds, so 48 fits plus a final refit. Cross-validated scores range from 0.595 to 0.794, which shows how much the settings matter. The best combination is C=1 with unigrams and sublinear_tf=True.

Look at the last two numbers together: the best cross-validation score is 0.794, but accuracy on the untouched test set is 0.722. That gap is normal and useful. Choosing the best of twelve candidates slightly flatters the winner, and 54 test documents is a small sample, so one or two documents move the score by several points. The golden rule: tune with cross-validation on the training data, and touch the test set once, at the end.

Habits that pay off

  • Name your steps clearly. The names become the prefixes in step__param and in cv_results_.
  • Save the pipeline, not the model. Persisting the whole object (for example with joblib) guarantees that production applies the same preprocessing as training.
  • Keep grids small at first. Cost grows multiplicatively with each parameter. RandomizedSearchCV samples a fixed number of combinations from larger spaces.
  • Pass raw text in. If your pipeline needs custom cleaning, wrap it in a function and give it to the vectorizer's preprocessor or tokenizer, so it is part of the pipeline too.
  • Compare against a baseline. Run the same cross-validation for a trivial model (for example DummyClassifier) so you know what "better" means.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main way a Pipeline prevents data leakage?

    When the whole pipeline is cross-validated, each fold fits the vectorizer and any selectors on its own training data, so nothing is learned from the validation fold.
  2. 2. In GridSearchCV, what does the parameter name clf__C refer to?

    Double underscores walk down the pipeline, so clf__C is the C hyperparameter of the step you named "clf".
  3. 3. Why is a TfidfVectorizer inside a ColumnTransformer given "title" while OneHotEncoder is given ["source"]?

    Text vectorizers iterate over documents, so they take a single column. Encoders and scalers work on 2D arrays, so they take a list of columns.
  4. 4. Your grid search reports a best CV score of 0.79 and your test accuracy is 0.72. What is the most reasonable reading?

    Selecting the maximum over many candidates is optimistic, and small test sets are noisy. The test score is the more honest number, as long as you did not tune on it.
  5. 5. In the leakage demo, why does feature selection on all the data give about 0.82 on random labels?

    With many words and few documents, some words correlate with random labels by chance. Selecting them on all data lets the validation labels influence the choice, which inflates the score.

Summary

  • A Pipeline chains transformers and an estimator into one object with fit, predict and score, working on raw text.
  • Cross-validate the whole pipeline so every step is refitted inside each fold; this prevents leakage, which was enormous in our random-label example (0.817 leaky versus 0.517 honest).
  • ColumnTransformer applies a different transformer to each column: give text vectorizers a column name and other transformers a list.
  • GridSearchCV searches hyperparameters of all steps using step__param names, then refits the best combination.
  • Tune on training data with cross-validation and evaluate on the test set only once; save the whole pipeline for production.

Next, learn which numbers to report in Evaluating Classifiers.