Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 63 ยท Sequences and Structure

Part-of-Speech Tagging

Learn POS tagging in Python: tag sets, a most-frequent-tag baseline, how a hidden Markov model resolves ambiguity, and Viterbi decoding written from scratch.

Up to now, we treated text as a bag of words: order was thrown away and every word was an independent feature. Language is more than a bag. In "I can fish" the word can is a helper verb and fish is a verb; in "a can of fish" both are nouns. Part-of-speech (POS) tagging labels every word with its grammatical category, so later steps know whether fish is something you eat or something you do.

POS tags are a basic ingredient of lemmatization (the lemma of saw depends on whether it is a noun or a verb), keyword filtering (keep nouns and adjectives), named entity recognition and syntactic parsing. The task is a first example of sequence labeling: the input is a sequence of words and the output is one label per word, where neighboring labels influence each other.

Tag sets

Different projects use different tag sets. Two are common:

  • The Penn Treebank tag set has about 45 fine-grained tags, such as NN (singular noun), NNS (plural noun), VBD (past-tense verb) and VBG (verb in -ing form). NLTK's default tagger uses it.
  • The Universal Dependencies (UPOS) tag set has 17 coarse tags that work across languages: NOUN, PROPN, VERB, AUX, ADJ, ADV, ADP, DET, PRON, CCONJ, SCONJ, NUM, PART, INTJ, PUNCT, SYM and X. spaCy exposes both: token.pos_ is UPOS and token.tag_ is the fine tag.

Open classes

NOUN dog, idea, river
VERB run, think, fish
ADJ small, green, old
ADV quickly, very, often

New words join these classes all the time.

1 / 2

Converting from fine tags to coarse ones is a simple table lookup, shown below on one sentence. Notice how several Penn tags collapse into one universal tag.

Python From Penn tags to universal tags

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why tagging is hard: ambiguity

Most words have one dominant tag. the is almost always a determiner, and river is almost always a noun. The trouble comes from the words that have several tags: can, fish, run, book, light, back. A large share of the running words in any text are ambiguous, even though most dictionary entries are not.

The simplest tagger ignores context entirely: give each word the tag it most often had in training, and tag unknown words as nouns. This most-frequent-tag baseline is surprisingly strong (around 90% on English news) and it is the yardstick any real tagger must beat. We try it on a tiny hand-tagged corpus built to include the ambiguous words can and fish.

Python The most-frequent-tag baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline gets 20 of 23 words right (87.0%). Its three mistakes are exactly the ambiguous words: fish always becomes a verb, because it was a verb and a noun equally often and the tie went to the first one seen, and can in a can becomes an auxiliary instead of a noun. A word-by-word lookup cannot know that after a the word fish or can has to be a noun, while after cat the word can is a helper verb. The tag depends on the neighbors.

The last two lines show the second weakness, unknown words. Calling every new word a noun is a decent default, because new words are mostly nouns, but a few suffix rules fix obvious cases: barked ends in -ed and loudly in -ly. Real taggers learn such rules from data, along with capitalization and word shape.

The hidden Markov model idea

A hidden Markov model (HMM) tagger uses context in the simplest possible way. It scores a whole tag sequence by multiplying two kinds of probabilities for each position:

  • the transition probability P(tag | previous tag): how likely a noun is after a determiner, or a verb after an auxiliary;
  • the emission probability P(word | tag): how likely the tag NOUN is to produce the word fish.
TEXT
P(words, tags) = product over positions of  P(tag_i | tag_(i-1)) * P(word_i | tag_i)

The tags are "hidden" because we only see the words and must infer the tags behind them. Both tables come from counting in a tagged corpus, with a little smoothing (here, adding 0.1 to every count) so that unseen combinations are unlikely but not impossible, exactly as in n-gram language models. A tag for each word is chosen to maximize the product for the whole sentence, not word by word.

Trying every tag sequence is impossible: 8 tags and 4 words gives 4,096 sequences, and a 30-word sentence gives 8^30. The Viterbi algorithm solves this with dynamic programming. At each position, for each tag, it keeps only the best-scoring path that ends there, and remembers which previous tag it came from (a backpointer). At the end, it picks the best final tag and follows the backpointers backwards. The work grows with sentence length times the square of the number of tags, not exponentially.

Viterbi in four steps

Start Score each tag for the first word.
Extend For each tag, keep the best previous tag.
Finish Pick the best tag at the last word.
Backtrack Follow the backpointers to the start.

Scores are log probabilities, so they are added, not multiplied.

1 / 2

We work with log probabilities, so products become sums and long sentences do not underflow to zero. Here is the full tagger, with both tables learned from the same toy corpus.

Python An HMM tagger with Viterbi decoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All 23 words are tagged correctly, including the three that defeated the baseline. The learned numbers explain why: after a determiner a noun is very likely (0.75) and a verb is almost impossible (0.004), so fish in a fish becomes a noun even though the word is equally common as a verb. And can is far more likely to be produced by an auxiliary (0.494) than by a noun (0.043), but in the man sees a can the transition from DET to NOUN outweighs it.

The last block shows Viterbi's bookkeeping for a fish can swim. After fish, the best path ending in NOUN (-4.4) is much better than the best path ending in VERB (-7.8); the verb reading is not discarded forever, it is just worse. The scores are negative because they are logs of probabilities below 1; higher (closer to zero) is better.

Our toy corpus has 17 sentences, so it is easy to be perfect. A real HMM tagger trained on a million words reaches about 95 to 96% on English, mostly because of unknown words. The HMM also has a built-in limit: it only sees the previous tag and the current word. Section Sequence Labeling with CRFs and Features shows how to add suffixes, capitalization and neighboring words as features.

Using real taggers

You rarely train a tagger by hand. Libraries ship trained models, usually neural networks, with accuracy around 97% on news text. The code below is read-only because the models are too large to download in the browser.

Python
# NLTK (Penn Treebank tags)
import nltk
nltk.download("averaged_perceptron_tagger_eng")
print(nltk.pos_tag("They can fish in the river".split()))  # list of (word, tag) pairs

# spaCy (UPOS in pos_, fine tag in tag_)
import spacy
nlp = spacy.load("en_core_web_sm")
for token in nlp("They can fish in the river"):
    print(token.text, token.pos_, token.tag_)

Typical pitfalls are domain shift (a tagger trained on news does poorly on tweets or medical notes), sentence boundaries (split first, see sentence segmentation), and tokenization differences between your pipeline and the tagger's training data.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the most-frequent-tag baseline fail on "a fish can swim"?

    A word-by-word lookup gives fish the same tag everywhere. Only context (the preceding determiner) shows that fish is a noun here.
  2. 2. In an HMM tagger, what does the emission probability P(word | tag) express?

    Emissions link tags to words (for example how often NOUN produces the word fish). Transitions link tags to neighboring tags.
  3. 3. What does the Viterbi algorithm avoid?

    By keeping only the best path into each tag at each position (plus a backpointer), Viterbi finds the best whole sequence in time proportional to length times the square of the tag count.
  4. 4. Why do implementations add log probabilities instead of multiplying probabilities?

    A long sentence multiplies many numbers below 1, which quickly underflows. Summing logs gives the same ordering of paths without that problem.
  5. 5. What is the purpose of smoothing the transition and emission counts?

    Without smoothing, one unseen combination makes a whole path impossible (probability zero). Adding a small count keeps it unlikely but possible.

Summary

  • POS tagging labels each word with a grammatical category, using a fine tag set (Penn Treebank) or a coarse universal one (UPOS).
  • Ambiguous words such as can and fish make context necessary; the most-frequent-tag baseline is strong but cannot use it.
  • An HMM scores a tag sequence with transition probabilities P(tag | previous tag) and emission probabilities P(word | tag), both estimated by smoothed counting.
  • Viterbi decoding finds the best whole sequence with dynamic programming, log scores and backpointers.
  • Unknown words are the main weakness; suffix and shape features help.
  • Pre-trained NLTK and spaCy taggers are the practical choice; always compare against the baseline.

Next, find the names of people, places and organizations with Named Entity Recognition.