Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 63 ยท Counting and Classic Representations

Word Frequencies and Zipf's Law

Count words with Counter, plot a rank-frequency curve with matplotlib, see Zipf's law in real text, and measure how vocabulary grows with Heaps' law.

The simplest thing you can do with text is count words. It sounds too basic to matter, but word counts power search engines, keyword extraction, spam filters and the first baselines of almost every NLP project. Counting also reveals a surprising regularity: in every large body of text, a handful of words are used all the time while most words are rare, and the way the numbers fall follows a pattern.

That pattern is called Zipf's law. In this lesson you will count words with Counter, plot the famous rank-frequency curve with matplotlib, fit a line to it, and then look at a second regularity, Heaps' law, which describes how fast the vocabulary grows as you read more text. You will need the tokenizer from tokenization basics; here we use a simple regular expression.

A real corpus without any files

The browser runner has no internet and no files, so we need text that is already in memory. Python ships with a lot of English prose in the form of documentation strings: every module, class and function can have a docstring. We collect the docstrings of thirty standard-library modules and treat them as one corpus. It is technical English rather than novels or news, but it is real, written by many people, and big enough for the statistics to behave.

Counting words, step by step

Tokenize Lowercase, then split into words.
Count Counter maps each word to its count.
Rank Sort from most to least frequent.
Plot Rank on x, frequency on y, both on log axes.
1 / 2
Python Counting words with Counter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see a corpus of roughly 28,000 tokens made of about 3,000 distinct words. These two numbers have names: tokens are running words (every occurrence counts) and types are distinct words. The top of the table is dominated by the, a, is, to and of, all function words, which is why stop-word lists exist. The last two lines show the lopsided shape of language: more than a third of the vocabulary appears only once (such words are called hapax legomena), while the 100 most common words cover about half of all tokens.

Zipf's law

George Kingsley Zipf noticed in the 1930s that if you sort words by frequency, the frequency of a word is roughly inversely proportional to its rank:

frequency(rank) โ‰ˆ C / rank

The most common word occurs about twice as often as the second, three times as often as the third, and so on. Equivalently, rank ร— frequency is roughly constant. It is a rule of thumb, not a physical law, but it holds across languages, genres and even for things that are not words, such as city sizes.

On ordinary axes the curve is a steep cliff followed by a long flat tail, which is hard to read. On log-log axes a power law becomes a straight line with slope near -1, so that is how Zipf curves are drawn.

Python The rank-frequency plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted slope comes out near -1 (about -0.9), and the dots follow the ideal line for the first few hundred ranks. Look at the table too: rank ร— frequency is 1,613 at rank 1 and between 3,000 and 4,200 at ranks 10 to 1,000. It does not stay perfectly constant, and that is expected. Two deviations are typical. At the far right the dots form horizontal steps, because many words share the same small count (a word cannot occur 1.5 times). At the far left the very top words sit slightly below the ideal line.

The exact numbers depend on your corpus. A novel, a news archive and this docstring corpus all give slopes near -1, but never exactly the same values, so treat Zipf's law as "a power law with exponent around one".

Why Zipf's law matters for NLP

The shape of this curve explains many design choices you will meet in later lessons:

  • A few words dominate the counts. That is why raw counts hide the interesting words, and why TF-IDF down-weights words that appear everywhere.
  • Most words are rare. A model trained on a sample will meet new words in new text. This is the out-of-vocabulary problem solved by subword tokenization.
  • The long tail is huge. Rare events, taken together, are common. Even if each rare word is unlikely, some rare word appears in almost every document, which leads to sparse data and the need for smoothing in language models.
  • Small vocabularies go a long way. Keeping only the top few thousand words captures most tokens, which is why vocabulary caps work in practice.

Heaps' law: vocabulary growth

Zipf's law describes one corpus at one moment. Heaps' law describes how the number of distinct words V grows with the number of tokens n as you read more:

V(n) โ‰ˆ K ร— n^ฮฒ

For English ฮฒ is typically between 0.4 and 0.6. The key consequence is that the vocabulary never stops growing: a bigger corpus always contains new words, just more slowly. Doubling the amount of text does not double the vocabulary, but it does keep adding some.

Python Vocabulary growth and Heaps' law

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted exponent is about 0.6, close to the range typical for English, and the three checkpoints show the fit tracking the real growth. The straight line on log-log axes is the signature of a power law again. The documents are shuffled with a fixed seed on purpose: if topics arrived in blocks, the curve would jump whenever a new topic began, so always shuffle before measuring vocabulary growth.

Practical uses of counts

  • Choose a vocabulary size. Plot cumulative coverage (the share of tokens covered by the top k words) and pick the k where the curve flattens.
  • Find suspicious tokens. Very frequent tokens that are not words (nbsp, br) reveal cleaning problems; see cleaning and normalizing text.
  • Compare corpora. Words that are much more frequent in one corpus than another characterize it; this idea returns in bag of words and n-grams.
  • Spot-check tokenizer changes. If a change moves the number of types dramatically, look at why.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does Zipf's law say about word frequencies?

    The word at rank 2 appears about half as often as the word at rank 1, rank 3 about a third, and so on.
  2. 2. Why are rank-frequency plots drawn on log-log axes?

    Taking logarithms of both axes turns frequency = C / rank into a straight line, which is easy to judge by eye.
  3. 3. What is the difference between tokens and types?

    A text of 28,000 tokens may contain only 3,000 types, because common words repeat.
  4. 4. What does Heaps' law imply when you double the size of a corpus?

    Vocabulary grows as K times n to the power beta with beta below 1, so it keeps growing but more slowly than the text.
  5. 5. Why does Zipf's law make out-of-vocabulary words unavoidable?

    The long tail of rare words means any fixed vocabulary will miss words in new text.

Summary

  • Counting words with Counter gives tokens (all occurrences) and types (distinct words).
  • Zipf's law: frequency is about C / rank, a straight line with slope near -1 on log-log axes.
  • A few words cover much of any text, and a large share of the vocabulary appears only once.
  • Heaps' law: vocabulary grows like K ร— n^ฮฒ with ฮฒ around 0.5, and never stops growing.
  • These shapes motivate stop words, TF-IDF weighting, smoothing and subword vocabularies.

Next, count pairs and triples of words in N-grams.