Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Inference and Optimization

Decoding and Sampling

Learn how a language model turns logits into text: greedy decoding, temperature, top-k, top-p (nucleus) sampling, repetition penalties and beam search, with NumPy code.

A language model does not output words. At each step it outputs a vector of logits, one number per vocabulary token. Decoding is the procedure that turns those logits into the next token, appends it to the text, and repeats. The choice of decoding strategy changes the output as much as many model changes do: the same model can be dull and repetitive, creative, or incoherent depending on a few settings.

flowchart LR
    A["Prompt tokens"] --> B["Model forward pass"]
    B --> C["Logits (one per token)"]
    C --> D["Adjust: temperature, penalties, filters"]
    D --> E["Pick next token"]
    E --> F["Append and repeat"]
    F --> B

Everything in this lesson happens in boxes D and E.

From logits to probabilities

The softmax turns logits z into probabilities: p_i = exp(z_i) / sum_j exp(z_j). Temperature T divides the logits first: softmax(z / T).

  • T < 1 sharpens the distribution: likely tokens become more likely.
  • T = 1 is the model's own distribution.
  • T > 1 flattens it: unlikely tokens get more chance.
  • T -> 0 approaches greedy decoding.
Python Temperature reshapes the distribution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At T=0.2 almost all of the mass sits on "the". At T=3.0 even "quantum" gets a real share.

Greedy decoding

Greedy takes the argmax every step. It is deterministic and fast, and good for tasks with one right answer such as extraction or short classification. For open-ended text it tends to loop ("I think that I think that...") because the most probable continuation of a repetitive text is more repetition.

Sampling and truncation

Sampling draws the next token at random from the (adjusted) distribution. Pure sampling at T=1 has a flaw: the vocabulary has tens of thousands of tokens, and the long tail of junk tokens together holds a noticeable probability. Over hundreds of steps you will draw some of them. So we truncate the tail:

  • Top-k: keep only the k most likely tokens, renormalize.
  • Top-p (nucleus): keep the smallest set of tokens whose cumulative probability reaches p, renormalize. The set size adapts: small when the model is confident, large when it is unsure.
  • Min-p: keep tokens with probability at least min_p times the top token's probability.
Python Top-k and top-p filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tokens like "banana" and "quantum" get probability zero after truncation, so they can never be sampled.

Typical defaults for chat: T around 0.7, top_p around 0.9. For code or math: low temperature (0 to 0.3). For brainstorming: higher temperature. Set one of top_k or top_p as the main filter rather than tuning several at once.

Repetition control

Models repeat. Mitigations modify logits before sampling:

  • Repetition penalty: divide (or subtract from) the logit of tokens already seen.
  • Frequency and presence penalties: subtract a * count and a constant for any token that appeared.
  • No-repeat n-gram: forbid generating any n-gram that already occurred.

Too much penalty hurts: it also penalizes words that must repeat, such as names and code identifiers.

Beam search keeps the B best partial sequences by total log-probability, expanding each by every token and pruning back to B. It searches for a high-probability whole sequence rather than a locally greedy one. It suits translation and summarization, where there is a fairly well-defined target, and it is rarely used for chat, because maximum-probability text is bland and often repetitive.

Python Beam search beats greedy on a toy model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy picks "A" first (0.5 beats 0.4), then "" (0.7): total probability 0.35. Beam search keeps "B" alive too and finds B, C, end with 0.4 x 0.9 x 1.0 = 0.36, a slightly more probable whole sequence that greedy can never reach because it committed to "A" at step one. Real models have far more such forks, so beam search can raise sequence probability, though higher probability does not always mean better text.

Practical settings

Task Setting
Extraction, classification, SQL temperature 0, or greedy
Chat temperature 0.6 to 0.8, top-p 0.9
Creative writing temperature 0.9 to 1.2, top-p 0.95
Reasoning with voting temperature around 0.7, several samples (self-consistency)
Translation beam search, width 4 to 5

Also set a maximum number of new tokens and stop sequences, or the model may run on. Note that even temperature 0 is not guaranteed to be bit-for-bit deterministic across hardware and batch sizes, because floating-point sums can be reordered (serving).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does lowering the temperature do?

    Dividing logits by T < 1 makes differences larger, so the top tokens get more probability.
  2. 2. Why is top-p often preferred to a fixed top-k?

    When one token dominates, the nucleus is tiny; when the model is unsure, the nucleus widens.
  3. 3. Why does greedy decoding often loop on open-ended text?

    Once a pattern repeats, the model assigns high probability to repeating it again, and argmax follows that forever.
  4. 4. Beam search is most appropriate for which task?

    Translation has a well-defined target, so maximizing sequence probability works; chat prefers diversity.

Summary

  • Decoding converts logits to tokens: greedy, sampling, or beam search.
  • Temperature rescales logits; top-k, top-p and min-p remove the unlikely tail.
  • Repetition penalties help but can hurt names and code.
  • Match settings to the task: low temperature for exactness, higher for creativity.

Next: The KV Cache