Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 49 ยท Computational Linguistics

Chapter overview: Computational linguistics

A tour of computational linguistics and natural language processing: the pipeline from text to meaning, the history from rules to neural models, and what this chapter covers.

Ask your phone for directions, watch a video with automatic captions, or read a web page translated in a second: you are using computational linguistics. The field sits where linguistics meets computer science and statistics. It has two goals that overlap: to build useful language technology (called natural language processing, NLP) and to use computational models to test ideas about how language works.

Five cards in a row: text, tokens, tags, parse and meaning, with two pills contrasting rule-based and statistical methods
A classic pipeline turns raw text into structure and then meaning. Modern systems often learn several of these steps at once.

What you will learn

  • The classic steps from raw text to meaning
  • How the field moved from hand-written rules to statistics to neural models
  • What makes language hard for computers
  • What the lessons in this chapter will cover

Key terms

Corpus: a large, structured collection of texts used for study and training.

Token: a unit of text such as a word or punctuation mark.

Model: a mathematical description that predicts or labels language.

Ambiguity: a string with more than one possible analysis, the central difficulty of NLP.

Evaluation: measuring how well a system performs on data it has not seen.

From text to meaning

A typical pipeline processes Cats chase mice. in stages:

  1. Tokenization splits the string into tokens: Cats | chase | mice | .
  2. Tagging labels each token with a part of speech: NOUN VERB NOUN PUNCT.
  3. Parsing builds a structure: a sentence made of a noun phrase and a verb phrase (as in the syntax chapter).
  4. Semantic analysis extracts who did what to whom: chase(cats, mice).

Every stage can go wrong, and mistakes pass downstream. Time flies like an arrow has several parses (is flies a verb or a noun?) and each has a different meaning; picking the right one needs knowledge about the world and about typical language use.

A short history

  • 1954: The Georgetown-IBM experiment translated more than sixty Russian sentences into English with a tiny vocabulary and a handful of hand-written rules. Its organizers predicted machine translation would be solved in a few years; it was not.
  • 1966: Joseph Weizenbaum's ELIZA imitated a therapist using simple pattern matching. People were drawn into conversations with it, although it understood nothing. This effect is a warning about attributing understanding to a system that merely produces fluent text.
  • 1960s to 1980s: The era of hand-built grammars and knowledge bases. Rules were precise but brittle.
  • Late 1980s to 2000s: The statistical turn. Instead of writing rules, researchers learned patterns from corpora such as the Brown Corpus and the Penn Treebank. IBM's translation models of the early 1990s learned from aligned parallel texts. A frequently quoted remark, attributed to Frederick Jelinek of IBM, was that every time he fired a linguist, the speech recognizer got better. (The remark is a joke and an oversimplification; linguistic insight remained important for designing models.)
  • 2010s onward: Neural models. Word vectors (Mikolov and colleagues, 2013) represent words as points in space, and the Transformer architecture (Vaswani and colleagues, 2017) underlies today's large language models trained on vast amounts of text.

Rules

Hand-written precise but brittle
ELIZA, 1966 pattern matching
1 / 4

What this chapter covers

Lesson What it covers
Corpora and Zipf's law Counting words, frequency distributions, why most words are rare
n-gram language models Predicting the next word, smoothing, perplexity
Tagging and parsing Part-of-speech tagging and the CKY chart parser
Chapter review One picture and a ten-question quiz

Common misconceptions

"If a system writes fluent text, it understands language like we do." ELIZA shows how little is needed to look understanding. Whether modern large models understand is debated; evidence from careful tests is mixed.

"Statistical methods replaced linguistics." Annotated corpora, evaluation categories and model design all rely on linguistic knowledge, and linguistic analysis is used to diagnose model failures.

"Models are neutral." Models learn the patterns in their training data, including social biases.

Try it

The first demo is a minimal pipeline: it tokenizes a sentence with a regular expression and tags the tokens from a small lookup dictionary. Real taggers handle ambiguity; this one does not, which you can see by feeding it flies.

Python Tokenize and tag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second demo is an ELIZA-style responder with four pattern rules. It matches text against patterns and echoes pieces back, with no understanding at all.

JS A tiny ELIZA

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a corpus?

    Corpora supply the data for counting, training and testing.
  2. 2. Which step comes right after tokenization in the classic pipeline?

    Tagging assigns categories to the tokens, and parsing then uses them.
  3. 3. What did ELIZA demonstrate?

    People attributed understanding to a program that only matched patterns.
  4. 4. What marked the statistical turn in NLP?

    Models trained on data became more robust than purely hand-written systems.
  5. 5. Why is ambiguity central to NLP?

    Each stage must resolve ambiguity using context.

Summary

  • Computational linguistics builds language technology and tests linguistic ideas.
  • The classic pipeline runs from tokens to tags to parses to meaning.
  • The field moved from rules to statistics to neural models; each stage kept some of the lessons of the last.
  • Ambiguity, evaluation and bias are permanent themes.

Next: Corpora and Zipf's law