Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 53 ยท Sampling and Decoding

Top-k, Top-p and Min-p Sampling

Learn how top-k, top-p (nucleus) and min-p sampling cut off unlikely tokens, how to combine them with temperature, and how seeds make runs repeatable.

Temperature reshapes the whole probability distribution, but it never removes anything. Even at a moderate temperature, a vocabulary of 100,000 tokens has a long tail of nonsense tokens, each with a tiny probability. Add up thousands of tiny chances and, sooner or later, one of them gets sampled and derails the sentence.

Truncation samplers fix this by cutting the tail before sampling. Top-k, top-p and min-p are three ways to decide where to cut. They build on softmax and temperature, so read that lesson first.

The pipeline

Most inference engines apply the steps in a fixed order: divide logits by temperature, convert to probabilities, drop tokens with a filter, renormalize what remains so it sums to 1, then sample.

flowchart LR
    L["Logits"] --> T["Temperature"]
    T --> S["Softmax"]
    S --> F["Filters:<br/>top-k, top-p, min-p"]
    F --> R["Renormalize"]
    R --> D["Sample one token"]

Filters remove unlikely tokens; the survivors are rescaled and one is drawn.

The exact order differs between libraries (some apply temperature after the filters), so check the documentation of the engine you use.

Top-k: keep the k best

Top-k keeps only the k highest-probability tokens. With k = 1 it is greedy decoding. With k = 50 the model chooses among the 50 best candidates. The weakness is that k is fixed: if the model is certain (one token has 95%), keeping 50 still lets junk in. If the model is truly unsure (many reasonable continuations), 50 may cut good options.

Top-p: the nucleus

Top-p, also called nucleus sampling, adapts the cut to the shape of the distribution. Sort tokens by probability, then keep the smallest set whose probabilities add up to at least p. When the model is confident the nucleus may be one or two tokens; when it is unsure the nucleus can contain hundreds.

Min-p: relative to the leader

Min-p keeps every token whose probability is at least min_p times the probability of the top token. If the top token has 60% and min_p = 0.1, the threshold is 6%. If the top token has only 10%, the threshold falls to 1%. Like top-p it adapts to confidence, but it is simple and does not need sorting. It is available in many open-source runtimes; hosted APIs mostly expose only temperature, top-p and sometimes top-k.

Three ways to cut the tail

Top-k Keep a fixed number of tokens.
Top-p Keep tokens until their probability adds up to p.
Min-p Keep tokens near the leader's probability.

Top-p and min-p adapt to how confident the model is.

1 / 2

Implementing the filters

Each filter takes a probability vector and returns a new one with removed tokens set to zero and the rest renormalized.

Python Three filters on one distribution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Top-k with k = 3 kept three tokens regardless of their probabilities. Top-p with 0.8 stopped as soon as the running total reached 80%. Min-p with 0.25 used a threshold of 0.25 ร— 0.40 = 0.10, so tokens at 10% or more survive.

Confident versus unsure

The adaptive behavior is the whole point. Compare a peaked distribution with a flat one.

Python How many tokens survive?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Top-k always keeps 40 tokens, while top-p and min-p keep few tokens when the distribution is spiky and many when it is flat.

Combining filters and temperature

APIs usually let you set several at once, and they stack: each filter removes more tokens. A common sensible setup is a moderate temperature with top-p around 0.9 to 0.95. OpenAI-style and Anthropic-style guidance both suggest changing one of temperature or top-p at a time, because they overlap in effect and are hard to reason about together.

Here is a small sampler that chains temperature, top-k, top-p and min-p, and draws tokens using a seeded generator.

Python A configurable sampler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Seeds and reproducibility

Sampling uses random numbers. If you fix the seed, the sequence of random numbers is fixed, so the same logits and settings give the same tokens. Many APIs offer a seed parameter for this reason. It is useful for tests and for debugging a surprising answer.

Python Same seed, same text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two lines are identical; the third differs. On hosted models, a seed gives only "best effort" repeatability, since model versions and hardware can change between calls. For strict determinism use greedy decoding and still expect rare differences.

Do not combine aggressive settings blindly. A tiny top-p with a very high temperature can still produce odd text, and top-k set to 1 makes temperature and top-p irrelevant.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main weakness of top-k compared with top-p?

    A fixed k lets junk in when the model is sure and may cut good options when it is unsure. Top-p adapts to the shape of the distribution.
  2. 2. Probabilities are 0.5, 0.3, 0.1, 0.06, 0.04. With top-p = 0.85, which tokens are kept?

    0.5 + 0.3 = 0.8 is below 0.85, so the third token (0.1) is needed to reach 0.9.
  3. 3. The top token has probability 0.40 and min_p is 0.25. What is the minimum probability a token needs to survive?

    Min-p multiplies the top probability by the ratio: 0.25 x 0.40 = 0.10.
  4. 4. Why do filters renormalize the remaining probabilities?

    After tokens are removed the remaining probabilities sum to less than 1, so they are rescaled.

Summary

  • Truncation samplers cut the unlikely tail before sampling to avoid derailing text.
  • Top-k keeps a fixed count; top-p keeps the smallest set reaching probability p; min-p keeps tokens close to the leader's probability.
  • Top-p and min-p adapt to the model's confidence.
  • Filters stack with temperature; change one main knob at a time.
  • A fixed seed makes sampling repeatable, though hosted models only promise best effort.

Next: Greedy Decoding, Beam Search and Repetition Penalties.