Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 49 ยท Computational Linguistics

Corpora and Zipf's law

Learn how linguists count words in corpora, what types, tokens and hapax legomena are, and why Zipf's law means a few words dominate while most words are rare.

Open any book and count the words. A few tiny words (the, of, and) will fill a large part of the page, while most of the distinct words will appear once or twice. This lopsided pattern holds in English, Chinese, Swahili and almost every text ever examined. It shapes how every language technology works, and it was first described systematically by a Harvard linguist named George Kingsley Zipf.

Two panels showing a steep rank-frequency curve on linear axes and a straight line on log-log axes
Zipf's law: frequency falls roughly as one over rank. Log-log axes turn the steep curve into a line.

What you will learn

  • What a corpus is and how types differ from tokens
  • What Zipf's law says and how to check it
  • Why the long tail of rare words causes the sparse-data problem
  • How frequency counts support concordances and collocations

Key terms

Token: one occurrence of a word. Type: a distinct word form.

Hapax legomenon (plural hapax legomena): a type that occurs exactly once in a corpus.

Type-token ratio: types divided by tokens, a rough measure of vocabulary variety.

Zipf's law: frequency is roughly inversely proportional to frequency rank.

Concordance: a list of all occurrences of a word with the words around it.

Corpora

A corpus (plural corpora) is a body of real language, chosen and stored for study. The first widely used electronic corpus was the Brown Corpus, compiled at Brown University in the early 1960s by W. Nelson Francis and Henry Kuฤera: one million words of American English written in 1961, drawn from 15 genres such as news, fiction and science writing. Later corpora grew enormously: the British National Corpus (about 100 million words), the Corpus of Contemporary American English, and web-scale collections of billions of words.

Three questions decide whether a corpus is useful:

  1. Representativeness. Does it reflect the language you want to study? A corpus of news text says little about text messages.
  2. Balance. Does it cover varied genres, or one source?
  3. Annotation. Is it tagged (parts of speech, syntax, named entities)? Annotated corpora such as the Penn Treebank, with syntactic trees for about a million words of text, trained the first statistical parsers.

Counting

Given the sentence the cat saw the dog, we have 5 tokens and 4 types (the counts once as a type, twice as a token). The type-token ratio is 4/5. Ratios depend heavily on text length: long texts repeat words, so the ratio falls as the corpus grows.

Frequency tables are the simplest output of a corpus. The top of an English table is full of grammatical words: the, of, and, to, a, in. Content words appear far below. This is what makes frequency lists useful for teaching: a learner who knows the most frequent 2,000 word families can already follow a large share of everyday text, though the exact coverage depends on genre.

Zipf's law

In Human Behavior and the Principle of Least Effort (1949), Zipf noticed that if you rank words by frequency, the product of rank and frequency is roughly constant:

frequency โ‰ˆ C / rank

So the second most frequent word occurs about half as often as the first, the third about a third, and so on. More generally, f โ‰ˆ C / rank^a with a near 1. On a plot of log frequency against log rank, the points fall near a straight line with slope about โˆ’1.

The law is approximate; the fit is better in the middle than at the extremes, and Benoit Mandelbrot proposed a refinement, f = P / (rank + B)^a, that fits real data better. Many other phenomena show similar heavy-tailed distributions: city sizes, income, website visits.

Why does Zipf's law happen?

Zipf's own explanation was a principle of least effort balancing speaker and listener. But in 1957 the psychologist George Miller observed that a monkey typing randomly, with some probability of hitting the space bar, produces "words" whose frequencies also follow a Zipf-like law. This does not prove that language is random; it shows that the law alone is weak evidence for any specific theory. Current explanations involve communicative efficiency, growth processes ("rich get richer"), and the structure of meaning, and the debate is not settled. You will test Miller's observation in the second demo.

The long tail and sparse data

Zipf's law has a hard consequence. In a corpus of a million words, roughly half of the types are hapax legomena, a number that varies with corpus size and genre. Many perfectly good words, and almost all long phrases, simply never occur in any given sample. A model that assigns probability zero to unseen events fails on new text. This sparse-data problem motivates smoothing, the topic of the next lesson.

Concordances and collocations

A concordance (or KWIC, "keyword in context") shows every occurrence of a word with its surrounding words, helping lexicographers see how a word is really used. Collocation describes words that occur together more often than chance: strong tea (not powerful tea), heavy rain, make a decision. Statistical measures such as pointwise mutual information rank word pairs by how surprising their co-occurrence is.

Common misconceptions

"Zipf's law proves a deep property of language." Random text can show it, so a fit alone is weak evidence.

"More data solves sparsity." A bigger corpus has more types too; the long tail persists and a hapax share stays substantial.

"A corpus is neutral." Its sources, time and domain shape what you find. A model trained on one domain may fail in another.

Try it

The first demo counts words in a short original text: types, tokens, the most frequent words, and the share of words that occur once. A small text only hints at Zipf's law; try pasting a longer one.

Python Count words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second demo repeats Miller's monkey: it types random letters and spaces, counts the "words" and plots rank against frequency on log-log axes.

Python Miller's typing monkey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 6 questions

of 6 answered You scored / 6 ยท perfect!

  1. 1. How many tokens and types are in the sentence the cat saw the dog?

    The occurs twice, so there are five tokens and four distinct types.
  2. 2. According to Zipf's law, the second most frequent word occurs about:

    Frequency is roughly proportional to 1 divided by rank.
  3. 3. What is a hapax legomenon?

    Hapax legomena form a large share of types in any corpus.
  4. 4. What did Miller's typing-monkey argument show?

    So a good fit alone does not tell us why language has the pattern.
  5. 5. Why does the long tail of rare words matter for models?

    That is the sparse-data problem.
  6. 6. What does a KWIC concordance show?

    KWIC stands for keyword in context.

Summary

  • A corpus is a structured sample of language; its balance and annotation matter.
  • Types and tokens measure variety; a few words dominate and most are rare (Zipf's law).
  • The long tail causes the sparse-data problem, which smoothing addresses.
  • Concordances and collocations turn counts into insight about usage.

Next: n-gram language models