Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Bits and Surprise

Learn self-information, -log2 p: why rare events carry more bits, how bits, nats and hartleys relate, and why information adds up for independent events.

If I tell you the sun rose this morning, you learn almost nothing. If I tell you a specific lottery ticket won, you learn a lot. Shannon captured this with one formula: the information carried by an event of probability p is -log2(p) bits. Rare events are surprising and therefore informative; certain events carry zero information.

This lesson derives the formula from three simple requirements, practices the units, and shows why logarithms are the only sensible choice.

Self-information

The self-information (or surprisal) of an outcome x with probability p(x) is

TEXT
I(x) = log2(1 / p(x)) = -log2 p(x)      (measured in bits)

A fair coin flip has p = 1/2, so I = 1 bit. One of 8 equally likely options has p = 1/8, so I = 3 bits. A certain event has p = 1, so I = 0.

Python Surprisal of some events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the die: -log2(1/6) is about 2.585, not a whole number. Information in bits does not have to be an integer. A coder that could use fractional bits would, on average, spend exactly that. Practical codes round up, which is a topic of Source Coding.

Why a logarithm?

Three natural requirements pin the formula down:

  1. Rarer means more informative. I(p) must decrease as p grows.
  2. Certain means zero. I(1) = 0.
  3. Independent events add. If A and B are independent, learning both should give the sum of what each gives: I(p_A * p_B) = I(p_A) + I(p_B).

Requirement 3 turns a product of probabilities into a sum of information. The only continuous functions with that property are logarithms. The base only sets the unit.

flowchart LR
    P["probability p"] --> S["surprisal -log p"]
    S --> U["unit = base of the log"]
    U --> B["base 2: bits"]
    U --> N["base e: nats"]
    U --> H["base 10: hartleys"]

One formula, three units.

Python Independent events add

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two numbers match exactly. That additivity is why information costs scale with message length: a message of n independent symbols carries the sum of their individual surprisals.

Bits, nats and hartleys

The base of the logarithm chooses the unit.

Base Unit Natural when
2 bit (shannon, Sh) Digital hardware, coding
e nat Calculus, physics, machine learning
10 hartley (ban, dit) Decimal digits

Converting is a constant factor, because log_b(x) = ln(x) / ln(b). One nat is 1/ln(2) = 1.4427 bits; one hartley is log2(10) = 3.3219 bits.

Python Converting units

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this course we use bits and log2 unless we say otherwise. Machine-learning libraries usually report cross-entropy in nats, so you will convert sometimes (see KL Divergence and Cross-Entropy).

Information of a message

Suppose each letter of a four-letter alphabet A, B, C, D is drawn independently with probabilities 1/2, 1/4, 1/8, 1/8. The message ABAD has probability (1/2)(1/4)(1/2)(1/8) = 1/128, so it carries log2(128) = 7 bits. Add the per-symbol surprisals 1 + 2 + 1 + 3 and you get the same 7.

Python Information in a whole message

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

AAAA is cheap because the common letter repeats; DDDD is expensive because every symbol is a rare one. A good code gives short descriptions to likely messages, which is the whole point of compression.

Self-information is about one specific outcome. The average of self-information over all outcomes is a different object, the entropy, which we meet next. Do not call a single outcome's -log2 p "the entropy".

Common mistakes

  • Using log (natural) in Python and calling the answer bits. math.log is natural; use math.log2 for bits.
  • Taking log2(0). An impossible event has infinite surprisal; code must handle p = 0 explicitly.
  • Assuming a bit is the same as a binary digit. A binary digit stores up to one bit; a skewed digit carries less than one bit of information.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the self-information of an event with probability 1/16?

    -log2(1/16) = log2(16) = 4 bits.
  2. 2. Which property makes the logarithm the right choice for information?

    Probabilities of independent events multiply, and log turns products into sums.
  3. 3. How many bits is one nat?

    1 nat = 1/ln 2 = 1.4427 bits. (3.3219 is bits per hartley.)
  4. 4. An event has probability 1. Its self-information is

    A certain event teaches you nothing: -log2(1) = 0.

Summary

  • Self-information I(x) = -log2 p(x): rare events carry more bits, certain events carry none.
  • The log appears because independent events must add their information.
  • Base 2 gives bits, base e nats, base 10 hartleys; they differ by constant factors.
  • A message's information is the sum of its symbols' surprisals.

Next: A Probability Refresher