Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 53 ยท Sampling and Decoding

Logits, Softmax and Temperature

See how a language model turns raw scores (logits) into probabilities with softmax, and how temperature trades determinism for creativity.

At the end of its forward pass, a language model does not hand you a word. It hands you one raw number for every token in its vocabulary, often 32,000 to 250,000 of them. These numbers are called logits. A bigger logit means the model prefers that token, but logits are not probabilities: they can be negative and they do not add up to 1.

Turning logits into a choice happens in two steps. First, softmax converts them into probabilities. Then a decoding strategy picks a token. This lesson covers softmax and the most famous knob in every LLM API, temperature. The next lessons cover top-k and top-p and beam search.

From logits to probabilities

Softmax exponentiates each logit and divides by the sum of all the exponentials:

p_i = exp(z_i) / sum_j exp(z_j)

The result is positive, sums to 1, and keeps the order of the logits. Differences matter, not absolute values: adding 100 to every logit changes nothing.

Python Softmax by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Subtracting the maximum before exp is the standard trick. It leaves the result unchanged but keeps the numbers small enough to avoid overflow, as the last two lines show.

Temperature

Temperature T divides the logits before softmax: p = softmax(z / T).

  • T = 1 leaves the model's own distribution unchanged.
  • T < 1 makes large logits stand out even more. The distribution gets sharper, and the model becomes more predictable.
  • T > 1 shrinks the gaps. The distribution gets flatter and less likely tokens get a real chance.
  • T โ†’ 0 approaches always picking the top token (greedy decoding). Most APIs treat 0 as exactly that.
  • T โ†’ โˆž approaches a uniform random choice.

Temperature does not add knowledge. It only changes how boldly the model chooses among options it already considers possible.

One knob, three moods

Low Sharp and repeatable. Good for facts and code.
Medium The model's natural spread.
High Flat and surprising. Good for brainstorming.

Too high and text turns into nonsense.

1 / 2
Python Plot the effect of temperature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At low temperature the bar for "cat" towers over the rest; at 2.5 the five bars are much closer in height.

Entropy: measuring how undecided the model is

Entropy summarizes how spread out a distribution is: H = -sum p_i log2 p_i, measured in bits. It is 0 when one token has probability 1, and it is largest (log2 of the vocabulary size) when all tokens are equally likely. Entropy is a handy single number for "how unsure is the model at this step?". Some production systems log it as a confidence signal.

Python Entropy as temperature changes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Entropy rises smoothly with temperature and approaches the maximum as the distribution becomes uniform.

Sampling: determinism versus creativity

With probabilities in hand, the simplest strategy is to sample: draw a token at random, weighted by its probability. A low temperature gives nearly the same pick every time. A high temperature gives variety, and also a higher risk of odd or wrong tokens, because an unlikely token chosen early changes everything that follows.

Python Draw 1,000 samples per temperature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When should you use which setting?

  • Low (0 to 0.3): extraction, classification, code, math, anything with one right answer.
  • Medium (0.5 to 0.8): general chat and writing.
  • High (1.0 and above): brainstorming, poetry, names, diverse samples.

Even at temperature 0, outputs can differ slightly between runs on real services, because floating point math on parallel hardware is not perfectly reproducible. Never assume bit-for-bit determinism from a hosted model.

Temperature is only one decoding setting. Samplers usually combine it with truncation methods such as top-k and top-p, which are the topic of the next lesson.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are logits?

    Logits are raw scores. Softmax turns them into probabilities.
  2. 2. What happens to the distribution when temperature is lowered from 1 to 0.3?

    Dividing logits by a number below 1 widens the gaps between them, so the biggest probabilities grow.
  3. 3. Why subtract the maximum logit before computing exp in softmax?

    Softmax is unchanged by shifting all logits equally, and smaller exponents avoid overflow.
  4. 4. Which task suits a high temperature best?

    Brainstorming benefits from variety. The other tasks have one right answer, so low temperature is better.

Summary

  • Models output logits; softmax turns them into probabilities that sum to 1.
  • Temperature divides logits first: low is sharp and repeatable, high is flat and varied.
  • Entropy measures how undecided the model is.
  • Sampling draws tokens in proportion to probability; greedy decoding always takes the top one.
  • Use low temperature for exact tasks and higher for creative ones.

Next: Top-k, Top-p and Min-p Sampling.