Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 49 · Projects and Practice

Project: a language identifier

Build a working language identifier in Python from character trigram profiles, learn why it works, and test its limits on short, similar and mixed-language text.

How does your phone know that a message is in Spanish and not Portuguese? One of the simplest and most surprising answers uses no grammar and no dictionary: it counts character trigrams. Each language has its own habits of letter sequences, and a profile of those habits is enough to tell languages apart. In this project you build the whole thing, from counting to scoring, in about thirty lines of Python.

Four numbered steps: slice into trigrams, compare with profiles, add log-probabilities, pick the highest; and an example bar chart
A language identifier slices text into character trigrams and scores it against a profile for each language.

What you will learn

  • Why character n-grams carry language information
  • How to build a profile and score text with smoothed log-probabilities
  • How to test the identifier and see where it fails
  • What real systems do differently

Key terms

Character trigram: three consecutive characters, such as the or he_ (with a space).

Profile: counts of the trigrams seen in training text for one language.

Log-probability: the logarithm of a probability; adding logs replaces multiplying probabilities.

Smoothing: giving unseen trigrams a small probability so one gap does not rule out a language.

Baseline: a simple method against which you compare improvements.

The idea

Take the English word the. Its trigram the is very common in English text, he_ also, while in Spanish the sequence que and ción dominate, and German is full of sch, ein and ich. A language profile records how often each trigram occurs in a sample of the language. To identify a new text, ask which profile makes its trigrams most probable.

Concretely, for each language L we compute

score(L) = Σ log P(trigram | L)

over all trigrams in the text, with

P(trigram | L) = (count_L(trigram) + 1) / (total_L + V)

where V is the number of distinct trigrams across all training data (add-one smoothing, from the n-gram lesson). The language with the highest score wins. This is a naive Bayes classifier: it treats trigrams as independent, which is false but works surprisingly well. The method goes back to the 1990s: Ted Dunning (1994) described statistical language identification, and William Cavnar and John Trenkle (1994) used ranked n-gram profiles.

The project

You will train on four short, original paragraphs, in English, Spanish, German and French. Real systems train on millions of characters, so ours will be crude, but it will already work on easy cases. Run the first demo, then change the test sentences and try to fool it.

Python Train and classify

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Testing and failing

A classifier is only as good as its test. Try these experiments and note what happens:

  1. Very short text. Test a single word such as chat, which is a real word in both English and French. Few trigrams give little evidence.
  2. Close languages. Spanish versus Portuguese, or Norwegian versus Danish, share many trigrams. Real identifiers need more data and sometimes extra features.
  3. Mixed text. A tweet that switches between English and Spanish contains evidence for both; the winner is a poor description of the text. Code-switching, from the variation lesson, is a hard case.
  4. Names, numbers and URLs. They look the same in every language.
  5. Training size. Add more text to each profile and watch confidence grow.

The second demo shows how confident the model is by plotting the scores for one test sentence. Notice that the differences between languages are large when the text is long and shrink when it is short.

Python How confident is the classifier?

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What real systems do

Production identifiers such as Google's CLD3 or Facebook's fastText lid models use the same insight with refinements: much larger training data, character n-grams of several lengths, neural classifiers, and a "none of the above" option for text that matches no language well. They also handle scripts: Cyrillic, Greek and Chinese characters already narrow the choices before any n-gram counting. Language identification is a first step in nearly every multilingual pipeline: before you tag, parse or translate, you need to know what language you have.

Common misconceptions

"The classifier knows grammar." It knows only letter-sequence statistics. It cannot tell a sentence from a jumble of French-looking letters.

"More characters always means more accurate." Accuracy rises with text length but depends on the training data being similar to the test data.

"High accuracy on my small test means it works." Four sentences prove little. Measure on many held-out texts.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What is a character trigram?

    Trigrams slide over the text one character at a time.
  2. 2. Why do we add log-probabilities instead of multiplying probabilities?

    Long texts produce extremely small products that a computer cannot represent.
  3. 3. What problem does add-one smoothing solve here?

    Every trigram gets a small positive probability.
  4. 4. Which text is hardest for this classifier?

    Short text gives little evidence, and similar languages share trigrams.
  5. 5. Why is a four-sentence test not good evidence of accuracy?

    You need many held-out examples to estimate performance.

Summary

  • Character trigram profiles capture language-specific letter habits.
  • Scoring adds smoothed log-probabilities; the highest score wins.
  • Short, similar, mixed or named text are the weak spots.
  • Real systems use the same idea at scale, with more data and better models.

Next: Project: a sound-change simulator