Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 76 ยท Modelling

Working with Text Data

Turn text into numbers with tokenizing, bag of words and TF-IDF, then train and evaluate a support-ticket classifier with scikit-learn pipelines.

Much of the data a company owns is text: support tickets, reviews, emails, survey comments, product descriptions, chat logs. Models, however, work with numbers. The central task in text analysis is therefore turning words into columns in a way that keeps the useful signal. Once that is done, everything you already know about classification applies: split the data, train a model, evaluate it, look at the mistakes.

This lesson covers the classical, classroom-sized toolkit: tokenizing, bag of words, TF-IDF, and a simple classifier built in a scikit-learn pipeline. These methods are fast, cheap, easy to explain, and a strong baseline even in an age of large language models. They ignore word order and meaning, which is their main weakness; the end of the lesson says when to reach for something richer.

flowchart LR
    A["Raw text"] --> B["Clean and<br/>tokenize"]
    B --> C["Vocabulary:<br/>one column per word"]
    C --> D["Count or TF-IDF<br/>weights"]
    D --> E["Classifier"]
    E --> F["Predicted label"]

Text modelling turns each document into a row of word weights, then reuses ordinary classification tools.

Tokenizing and bag of words

Tokenizing splits a string into pieces, usually words. A typical first pass lowercases the text, splits on anything that is not a letter or digit, and optionally removes stop words (very common words such as "the" and "was" that carry little topic information). The result is a list of tokens per document.

In a bag of words each distinct token becomes a column, and each document becomes a row of counts. The word "bag" is meant literally: the words are tipped out of the sentence into a bag, so their order is lost. "The dog bit the man" and "The man bit the dog" produce the same row.

Python Bag of words for four tiny documents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three details to notice. The vocabulary is learned from the documents you pass to fit; later documents that contain unseen words simply ignore them. The matrix is mostly zeros, which is why scikit-learn stores it in a sparse format (toarray is only for display; on a real corpus it could use gigabytes). And n-grams, sequences of two or more words, put back a little of the lost word order ("not good" becomes its own column) at the cost of a much larger vocabulary.

TF-IDF: weighting words by how informative they are

Raw counts have a problem: frequent words get large numbers whether or not they are informative. TF-IDF (term frequency times inverse document frequency) fixes this with two ideas:

  • Term frequency (TF): a word that appears more often in a document matters more to that document.
  • Inverse document frequency (IDF): a word that appears in many documents tells you little, so its weight shrinks; a word found in only a few documents is distinctive and gets a boost.

Scikit-learn's TfidfVectorizer also normalizes each row to unit length, so long documents do not get larger numbers than short ones.

Python TF-IDF weights on the same documents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The word "delivery" appears in three of the four documents, so it has a low IDF and modest weights; "charged" and "twice" appear in only one document and carry large weights there. The effect is that rare, topic-specific words dominate each row. For many text problems TF-IDF is a better default than raw counts.

A support-ticket classifier

Now a realistic task: route short support tickets to the right team (billing, shipping or technical). We generate 500 synthetic tickets with a fixed seed so the results can be reproduced. Each ticket mixes common filler words, a few ambiguous words that appear in every category ("order", "account", "issue"), usually one or two words that really point to its team, and sometimes a word from another team's vocabulary. That makes the problem easy to start and impossible to get completely right, like real tickets.

Steps for a trustworthy evaluation:

  1. Split first, then fit the vectorizer on the training text only. The vocabulary and IDF values are learned from data; fitting them on all rows leaks information from the test set.
  2. Compare against a baseline (always predict the most common category).
  3. Put vectorizer and classifier into one Pipeline, so that cross-validation repeats both steps correctly.
Python Comparing vectorizers and classifiers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every model beats the baseline (always guessing the biggest category gives 0.44), and the simple TF-IDF plus logistic regression comes out on top at about 0.85. The bigram version is slightly worse: with only 350 training tickets, pairs of words are too rare to learn from, so the extra columns add noise. More features are not automatically better; the right amount depends on how much text you have.

Evaluating beyond accuracy

Accuracy hides which categories fail. The classification report and confusion matrix show it, and reading the words the model relies on tells you whether it learned something sensible. For a linear model the coefficients give the most influential words per class. The last step is error analysis: read some wrong predictions yourself.

Python Report, key words and error analysis

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several things to read from this output. Shipping is the easiest team (F1 0.89) and technical the hardest (0.79); technical is also the smallest class, so it has the least training text. The confusion matrix shows where the errors go: every team leaks a few tickets to each of the others, which is what you expect from tickets that mention another team's vocabulary. The top words per team are the topic words we planted, which is reassuring: the model learned the vocabulary, not an accident. The wrong tickets are what you would expect, too. "charge and courier is" (true team: technical) has no technical word at all, and "update for not after" (true team: shipping) contains a technical word and nothing else; no model could do better from that text. Finally, 5-fold cross-validation gives a mean accuracy of 0.78, with folds between 0.73 and 0.82, lower than the 0.85 of our single split. A single split of 150 test rows carries a few points of luck, so quote the cross-validated figure when you report results.

Always read a sample of misclassified texts. Errors that come from missing information tell you to collect better data; errors that come from clear words the model ignored tell you to improve the features.

Cleaning choices that matter

Preprocessing is a set of decisions, not a fixed recipe:

  • Lowercasing merges "Refund" and "refund". It is the default and almost always right, but it can hurt when case carries meaning (names, "US" versus "us").
  • Stop words are often worth removing for topic classification and harmful for tasks where small words matter ("not").
  • Stemming and lemmatizing reduce words to a base form ("charged", "charging" to "charge"). They shrink the vocabulary, but need extra libraries and language support.
  • Rare words can be dropped with min_df; extremely common words with max_df. Both reduce noise and size.
  • Numbers, URLs and emails may be better replaced with a placeholder token than treated as unique words.
  • Language. English rules do not work for Chinese (no spaces) or for morphologically rich languages. Check the tools you use.

Try each choice with cross-validation instead of arguing about it, and keep what helps.

When bag of words is not enough

Word counts ignore order, context and meaning. "Not good" and "good" share a column; "bank" the river and "bank" the institution share a column; synonyms such as "refund" and "reimbursement" are unrelated columns. Methods that go further include word and sentence embeddings (dense vectors in which similar meanings are close together), topic models, and transformer-based language models, which you can fine-tune or use for classification. They need more computing power, more data or a pre-trained model, and they are harder to explain. A sensible path is to build the TF-IDF baseline first. If it already solves the business problem, you are done; if not, you have a reference score the richer model has to beat. These ideas are covered in the machine learning and NLP tracks.

For the data side, remember the lessons from Cleaning Text and Categories: inconsistent spelling and encodings hurt a vocabulary-based model just as they hurt a group-by.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a bag-of-words representation, what is lost?

    Each document becomes a row of word counts. The words are tipped out of the sentence, so 'dog bites man' and 'man bites dog' look identical.
  2. 2. What does a low IDF value tell you about a word?

    IDF falls as the share of documents containing the word rises. Common words get small weights, rare words large ones.
  3. 3. Why should the vectorizer be fitted on the training text only?

    Vocabulary and IDF are learned from data. Learning them from rows that should be unseen makes the test score optimistic. A pipeline avoids the mistake.
  4. 4. A TF-IDF model with unigrams scores 0.85 and the same model with bigrams scores 0.81 on 350 training texts. A reasonable explanation is:

    With little text, most word pairs appear only once or twice, so the model cannot learn reliable weights for them.
  5. 5. What is the best use of the misclassified examples after training?

    Error analysis shows why the model fails and what to fix: the data collection, the labels or the features.

Summary

  • Text must become numbers: tokenize, build a vocabulary, and represent each document as a row of word counts (bag of words) or weights (TF-IDF).
  • TF-IDF down-weights words that appear in many documents and highlights distinctive ones.
  • N-grams recover a little word order but enlarge the vocabulary; with small corpora they can hurt.
  • Put the vectorizer and the classifier in one pipeline, fit on training text only, and compare with a baseline.
  • Look beyond accuracy: classification report, confusion matrix, influential words and a sample of wrong predictions.
  • Bag of words ignores order and meaning; build it as a baseline before moving to embeddings or language models.

Next: Explaining Model Behaviour, which shows how to look inside any model, text or not.