Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 63 ยท Cleaning and Tokenization

Tokenization: Splitting Text into Words

Learn word tokenization from first principles: whitespace and regex splitting, punctuation, contractions, hyphens, numbers, and a hand-written Python tokenizer.

A computer sees a text as one long string. Almost every NLP method needs it as a list of units first: the tokens. Tokenization is the step that decides what counts as one unit, and every later step (counting, tagging, classifying) inherits that decision. A tokenizer that splits "can't" into can and t, or glues "today," into one token, quietly damages everything downstream.

For English, words separated by spaces look like an easy case, and that is exactly why people underestimate tokenization. This lesson starts with the two obvious approaches, shows where they break, and then builds a small tokenizer that handles punctuation, contractions, hyphens, numbers, URLs and abbreviations, using only Python's re module.

Why tokenization is tricky

Punctuation Is "today," one token or two?
Contractions can't is can + not, not can + t.
Hyphens e-mail, well-known, state-of-the-art.
Numbers $4.99 and 1,299.50 are single things.

There is no single correct answer, only a good fit for the task.

1 / 2

Two simple baselines

The first baseline is text.split(), which cuts at whitespace. The second is a regex such as \w+, which pulls out runs of letters, digits and underscores. Compare them on a sentence that looks ordinary but is full of traps.

Python Whitespace, \w+ and word-or-punctuation splits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each baseline fails in its own way:

  • split() keeps punctuation glued to words: today,, e-mail:, (50% and off)! are not words. A search for off would miss off)!.
  • \w+ removes the punctuation but tears things apart: can't becomes can and t, U.S. becomes U and S, $4.99 becomes 4 and 99, and the email address splits into four fragments.
  • \w+|[^\w\s] keeps punctuation as separate tokens, which is often useful (a question mark can signal a question), but it still splits can't, the abbreviation and the price.

No single pattern is right yet. We need to say what should stay together.

Designing rules

Think about what units you want, and then write patterns for each kind, ordered from most specific to most general. With a regex, alternatives are tried left to right at each position, so URLs and emails must come before words, or the word pattern would grab the first letters of an address.

A workable set for English text:

  1. URLs: https?://\S+.
  2. Email addresses: letters, digits and a few symbols, then @, then a domain.
  3. Abbreviations: titles such as Dr. and dotted initials such as U.S., so the period stays inside the token.
  4. Numbers: optional currency sign, digits, optional decimal or thousands parts, optional percent sign.
  5. Words: letters, optionally joined by internal hyphens or apostrophes (e-mail, can't).
  6. Everything else: any single punctuation character as its own token.

In Python, named groups make the rules readable and let you ask which rule matched through match.lastgroup. The re.VERBOSE flag allows whitespace and layout inside the pattern. The word pattern [^\W\d_]+ means "letters only" (not non-word, not digit, not underscore), and works for accented letters and most scripts too.

Python A hand-written word tokenizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Now the sentence comes out as Dr., Smith, can't, visit, the, U.S., today, ,, e-mail, :, the full email address, costs, $4.99, (, 50%, off, ), !. Every token is a meaningful unit. The kind labels below show which rule fired for each token, which is useful for debugging and for later steps that treat numbers or URLs specially.

This tokenizer is still a first draft. Its abbreviation rule only knows a handful of titles and will misread a sentence-final period in some cases. For example, the "U.S." at the end of a sentence loses its sentence boundary. Sentence Segmentation deals with exactly that problem.

Contractions and clitics

Contractions such as don't, it's and we've join two words. Corpora like the Penn Treebank split them in a specific way: don't becomes do and n't, it's becomes it and 's. The motivation is that n't is a negation and 's is a separate grammatical piece, and downstream tools such as part-of-speech taggers expect them that way. For a bag-of-words classifier it can be better to keep don't whole, or even to expand it to do not.

The right choice depends on the consumer of the tokens. Here is a function that splits contractions the Treebank way, applied after the main tokenizer.

Python Splitting contractions (Treebank style)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The irregular table exists because the pieces of can't and won't are not simply can and wo. The second output list shows the Treebank-style result: ca and n't, it and 's, John and 's. Notice a limit: the rule cannot tell the possessive in John's from the contraction of is in it's; both are split the same way. Real tokenizers carry long lists of such special cases.

Hyphens, numbers and other scripts

Hyphens have no universal rule. In state-of-the-art the pieces are meaningful alone, in e-mail they are not, and in well-known either choice is defensible. Our tokenizer keeps hyphenated words whole, which keeps the vocabulary smaller but treats over-think and overthink as different words. If you prefer to split, change the word pattern to drop the [-'] part and handle apostrophes separately.

Numbers also vary by locale: 1,299.50 in English means one thousand two hundred ninety-nine and fifty hundredths, while some European languages write it as 1.299,50. Our rule accepts both separators inside a number but does not interpret them. Often a model does better if all numbers are replaced with a placeholder like <NUM>, because the exact value is rarely informative.

Finally, a warning about other writing systems. Whitespace tokenization works for English, French and many other languages, but Chinese and Japanese are written without spaces between words, and Thai too. The regexes above would return ไฝ ๅฅฝ as one long chunk. Languages like these need dictionary-based or learned segmenters, and modern multilingual models avoid the problem by tokenizing into subword pieces, which is the topic of Subword Tokenization.

Checking a tokenizer

A tokenizer is a program like any other, so test it. Collect a set of tricky sentences (abbreviations, prices, URLs, hyphens, quotes, emoji) and write down the tokens you expect. Rerun them every time you edit a pattern. You will use that idea again when you build a full, tested pipeline in Building a Reusable Preprocessing Pipeline.

Whatever tokenizer you choose, use the same one for training data, evaluation data and production text. Two tokenizers that disagree on a single rule can silently cost accuracy.

Standard tokenizer families

Most toolkits ship a few ready-made tokenizers. Knowing what they do helps you read their output and choose between them.

  • Whitespace tokenizer: splits on spaces only. Fast and predictable, but punctuation stays glued to words (panic:, $4.50,).
  • Word-punctuation tokenizer (NLTK calls it WordPunctTokenizer): a single regular expression, \w+|[^\w\s]+, that returns runs of word characters and runs of punctuation. It never fails, but it tears apart anything that contains punctuation: 4.50 becomes 4, ., 50, and e-mail becomes three tokens.
  • Treebank tokenizer (NLTK's TreebankWordTokenizer): a cascade of rules that follows the Penn Treebank conventions. It splits clitics (Don't becomes Do and n't), separates most punctuation, keeps 4.50 and e-mail whole and splits only a sentence-final period.
  • Sentence tokenizers (NLTK's PunktSentenceTokenizer) cut a text into sentences first. Punkt is unsupervised: it learns from a corpus which words are abbreviations and which words usually start sentences, instead of relying on a fixed list. The details are in Sentence Segmentation.
  • Model-based tokenizers (spaCy, Stanza, BPE or WordPiece in transformers) use trained rules or vocabularies, and are covered in Subword Tokenization.

The program below reproduces the first three behaviors with plain regular expressions so you can compare them on one sentence. The real NLTK classes are more thorough, but the contrast is the same.

Python Whitespace, word-punctuation and Treebank-style tokenizers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The whitespace output keeps panic: and [email protected]! as single tokens, which would be separate vocabulary entries in a counting model. The word-punctuation output is clean on punctuation but wrecks the price, the hyphenated word and the email address. The Treebank-style output is the usable one: Do, n't, $, 4.50, e-mail and the email address survive, while the colon, the comma and the exclamation mark are separated.

Tokenization challenges: phrases, compounds and domain terms

Splitting on characters is only part of the job. Three kinds of units need extra care.

  • Multi-word expressions (MWEs) are several words that act as one: New York, ice cream, kick the bucket, machine learning. Splitting them loses their meaning. A common fix is a dictionary of expressions that merges matching sequences into one token (NLTK ships MWETokenizer for this).
  • Compound words are written as a single word in languages such as German, Dutch and Swedish (Lebensversicherung, "life insurance") and sometimes in English (database). A search for insurance should find them, so these systems decompound: they split a word into known parts, often with a greedy longest-match against a dictionary or a statistical splitter.
  • Domain-specific terms do not follow everyday rules: chemical names (2-methylpropane), code identifiers (getUserName, max_retry_count), hashtags (#MachineLearning), part numbers and medical abbreviations. A general tokenizer treats them badly, so tokenizers for code, tweets or biomedical text add custom rules or are trained on that domain.
Python Merging phrases, splitting compounds and identifiers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The MWE step makes ice_cream and New_York single tokens, so a later frequency count treats them as the unit they are. The compound splitter finds bank, account, number in a word that has no spaces, and returns None for a word it cannot cover completely, which is the signal to keep the word whole. The last loop shows the standard trick for domain text: cut identifiers at underscores, hyphens and lower-to-upper case changes, so #MachineLearningRocks yields three searchable words.

Greedy longest-match decompounding can pick a wrong split (database + bank against data + base + bank). Real systems score the candidate splits, for example by word frequency, and keep the best one.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which problem does re.findall(r'\w+', text) cause for the string "can't"?

    The apostrophe is not a word character, so \w+ stops there and starts a new match, producing can and t.
  2. 2. In the alternation-based tokenizer, why must the URL and email patterns come before the word pattern?

    At each position the first alternative that matches wins. Specific patterns must precede general ones or they never get a chance.
  3. 3. How does the Penn Treebank convention tokenize "it's"?

    The convention splits clitics so that downstream taggers see the separate grammatical pieces, for example it and 's, do and n't.
  4. 4. Why do whitespace-based tokenizers fail for Chinese text?

    Languages written without spaces need dictionary-based or learned segmenters, or subword tokenization.
  5. 5. What is the best practice about the tokenizer used in training and in production?

    A mismatch changes the tokens the model sees and silently reduces accuracy.

Summary

  • Tokenization decides the units every later NLP step sees; split() leaves punctuation attached and \w+ tears apart contractions, prices and abbreviations.
  • A good tokenizer is an ordered alternation of patterns: URLs, emails, abbreviations, numbers, words with internal hyphens or apostrophes, then punctuation.
  • Named groups and lastgroup show which rule produced each token, and re.VERBOSE keeps the pattern readable.
  • Contractions can be kept whole, expanded or split Treebank-style; choose according to the downstream consumer.
  • Languages written without spaces need other methods, and every tokenizer should be tested on tricky sentences and used consistently.

Next: Sentence Segmentation