Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is Information Theory?

Meet information theory: how Claude Shannon turned 'information' into a number of bits, why it governs compression and reliable communication, and what this course covers.

Information theory is the mathematics of how much information a message carries and how reliably it can be stored or sent. It began with one paper in 1948, "A Mathematical Theory of Communication" by Claude Shannon, and it quietly sits underneath ZIP files, JPEG images, mobile phone networks, QR codes, DNA sequencing and modern machine learning.

The key move was to ignore meaning. A message is not valuable here because it is wise or funny; it is measured by how surprising it is among all the messages that could have been sent. That single idea turns a vague word, "information", into a number you can compute, compare and optimize. This lesson gives you the map of the whole course.

Everything in this course runs in your browser. The examples are plain Python (real Python 3.14 via Pyodide) using only the standard library, NumPy and SciPy. We build each idea from scratch first, then compare it with a library, so nothing is a black box.

The two big questions

Shannon asked two questions about a sender, a medium and a receiver.

  1. Compression. What is the smallest number of bits per symbol that lets you describe a source without losing anything? The answer is a quantity called entropy.
  2. Communication. What is the largest rate at which you can send bits through a noisy channel and still make the error probability as small as you like? The answer is the channel capacity.

Both answers are exact numbers, and both come with matching practical constructions. That is rare in engineering: Shannon proved where the wall is, and later people spent decades building codes that get close to it.

Information theory in one picture

Source Produces symbols
Compress Remove redundancy
Channel Adds noise
Protect Add smart redundancy

Compress first, then protect.

1 / 3

A first taste: twenty questions

Think of a secret number between 1 and 100. How many yes/no questions does a clever guesser need? Each answer splits the remaining candidates in half, so after k questions at most 2**k candidates can be told apart. You need 2**k >= 100, so k = 7. In other words, picking one of 100 equally likely options is worth log2(100) bits, about 6.64, and you pay in whole questions.

Python Binary search = counting bits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two columns agree. That is information theory in miniature: the amount of uncertainty in a situation, measured in bits, is the number of perfectly informative yes/no answers you need to remove it.

Predictable data is cheap

Now the same idea for data. A file that is highly predictable contains little information per byte and can be squeezed. A file of random bytes cannot, because every byte is a surprise.

Python Compressibility depends on predictability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two shrink dramatically, and the random bytes do not shrink at all (they even grow slightly, because of format overhead). No compressor can beat this: random data has nothing to remove. In Entropy you will compute exactly how many bits each source needs.

A short history

  • 1948: Shannon publishes the theory and coins the word "bit" (suggested by John Tukey).
  • 1950: Hamming invents the first practical error-correcting codes at Bell Labs.
  • 1952: Huffman, a student at MIT, finds the optimal prefix code.
  • 1970s-80s: arithmetic coding and the Lempel-Ziv family make compression practical for everyone.
  • 1993: turbo codes, later LDPC codes, approach Shannon's capacity limit, powering modern wireless and satellite links.

What you will learn

The course has nine chapters. You are in the first: vocabulary and probability. Then you will define entropy and its relatives (mutual information, KL divergence), prove why you cannot compress below entropy, build Huffman and arithmetic coders, study channel capacity, build error-correcting codes, see how real compressors like zlib and JPEG work, and finish with applications and projects.

You do not need advanced math. If you are comfortable with logarithms and basic probability, you are ready; the next lessons refresh both.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does information theory deliberately ignore when it measures information?

    Shannon measured how surprising a message is among the possible ones, not what it means to a human.
  2. 2. How many yes/no questions are needed, in the worst case, to find a number from 1 to 100?

    Each question halves the candidates, so you need the smallest k with 2^k >= 100, which is 7.
  3. 3. Which quantity is the minimum number of bits per symbol needed to describe a source without loss?

    Entropy is the compression limit; capacity is the communication limit.
  4. 4. Why does a file of random bytes not shrink under zlib?

    Compression removes predictability. Uniformly random bytes have none, so no lossless method can shrink them on average.

Summary

  • Information theory measures information as surprise, in bits, independent of meaning.
  • Two central numbers: entropy (best compression) and capacity (best reliable communication rate).
  • Halving the possibilities with each yes/no answer links bits to log2.
  • Predictable data compresses; random data does not.

Next: Bits and Surprise