Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 39 ยท The Microcanonical and Canonical Ensembles

Gibbs Entropy and Information

Generalize Boltzmann entropy to the Gibbs form S = -k_B sum p ln p, connect it to Shannon information and bits, and compute the Landauer limit kT ln 2.

Boltzmann's S = k_B ln ฮฉ applies when all ฮฉ microstates are equally likely. In the canonical ensemble the microstates have different probabilities. We need a more general entropy, and J. Willard Gibbs supplied it. Remarkably, the same formula appears in information theory, which links physics to computation.

The Gibbs entropy

For any probability distribution over microstates:

Text
S = โˆ’k_B ฮฃ_s p_s ln p_s

Because 0 โ‰ค p_s โ‰ค 1, each ln p_s โ‰ค 0, so S โ‰ฅ 0. A system certain to be in one microstate (p = 1) has S = 0.

Check against Boltzmann. In the microcanonical ensemble p_s = 1/ฮฉ for ฮฉ states:

Text
S = โˆ’k_B ฮฃ_s (1/ฮฉ) ln(1/ฮฉ) = โˆ’k_B ฮฉ (1/ฮฉ)(โˆ’ln ฮฉ) = k_B ln ฮฉ

The Gibbs form reduces to the Boltzmann formula. Moreover, among all distributions over ฮฉ states, the uniform one has the largest entropy; that is the content of the postulate of equal probabilities seen as "maximum uncertainty".

Python Gibbs entropy of different distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Entropy of the canonical ensemble

For the Boltzmann distribution p_s = e^(โˆ’ฮฒE_s)/Z we have ln p_s = โˆ’ฮฒE_s โˆ’ ln Z. Then

Text
S = โˆ’k_B ฮฃ p_s (โˆ’ฮฒE_s โˆ’ ln Z) = k_B ฮฒ โŸจEโŸฉ + k_B ln Z = โŸจEโŸฉ/T + k_B ln Z

Defining the Helmholtz free energy F = โˆ’k_B T ln Z (the subject of the next chapter), this reads S = (U โˆ’ F)/T, or F = U โˆ’ TS, the familiar thermodynamic relation. So the Gibbs entropy of the canonical distribution is exactly the thermodynamic entropy.

Python S from -sum p ln p equals U/T + k ln Z

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Connection to Shannon information

In 1948 Claude Shannon defined the information entropy of a message source, in bits:

Text
H = โˆ’ฮฃ p_i logโ‚‚ p_i

It measures the average number of yes/no questions needed to identify the outcome. Comparing with the Gibbs formula,

Text
S = k_B ln 2 ร— H

(one bit corresponds to k_B ln 2 โ‰ˆ 9.57 ร— 10^-24 J/K). So thermodynamic entropy is the missing information about the microstate, expressed in physical units. A gas whose microstate is unknown has huge entropy because identifying its microstate would take an enormous number of bits.

The Landauer limit

If information is physical, erasing it should cost something. Landauer's principle says that erasing one bit of information in an environment at temperature T must dissipate at least

Text
Q = k_B T ln 2

of heat. Erasing means resetting a bit with unknown value (entropy k_B ln 2) to a definite value (entropy 0); the entropy k_B ln 2 must go into the environment, carrying heat T ร— k_B ln 2. At 300 K this limit is about 2.87 ร— 10^-21 J, far below what present-day computers dissipate per operation, but it sets a fundamental floor.

Python Landauer limit at various temperatures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Landauer's bound is a statement about irreversible erasure. Reversible computation can in principle dissipate arbitrarily little, because it never throws away information.

Maximum entropy and the Boltzmann distribution

There is another way to see the canonical distribution. Maximize S = โˆ’k_B ฮฃ p_s ln p_s over all distributions with a fixed average energy โŸจEโŸฉ and total probability 1. Using Lagrange multipliers, the result is p_s โˆ e^(โˆ’ฮฒE_s), the Boltzmann distribution, where ฮฒ is the multiplier enforcing the energy constraint. The distribution is "the least biased guess consistent with what we know". This viewpoint, developed by E. T. Jaynes, is the basis of maximum-entropy inference in data science.

Python The Boltzmann distribution maximizes entropy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The optimizer's answer matches the Boltzmann distribution: equal ratios between neighboring levels, each equal to e^(โˆ’ฮฒ) because the levels are equally spaced.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The Gibbs entropy is:

    S = -k_B Sum p_s ln p_s works for any probability distribution and reduces to k_B ln Omega when p_s = 1/Omega.
  2. 2. A system has 8 equally likely microstates. What is its entropy in bits?

    log2(8) = 3 bits. In physical units this is 3 k_B ln 2.
  3. 3. What is the minimum heat dissipated to erase one bit at temperature T (Landauer)?

    Erasing one bit must export entropy k_B ln 2, which carries at least heat k_B T ln 2.
  4. 4. Which distribution maximizes the Gibbs entropy at fixed average energy?

    Maximizing S with constraints on normalization and mean energy yields p_s proportional to exp(-beta E_s).

Summary

  • Gibbs entropy: S = โˆ’k_B ฮฃ p_s ln p_s; it reduces to k_B ln ฮฉ for equal probabilities.
  • For the canonical ensemble S = U/T + k_B ln Z, equivalent to F = U โˆ’ TS.
  • S = (k_B ln 2) ร— (information in bits); thermodynamic entropy is missing information.
  • Landauer: erasing a bit costs at least k_B T ln 2 of heat.
  • The Boltzmann distribution is the maximum-entropy distribution for a given mean energy.

Next: The Partition Function