Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 38 ยท Pipelines and Inference

Text Generation and Decoding Strategies

Understand greedy search, beam search, temperature, top-k and top-p sampling, repetition penalties and stopping, with runnable NumPy demos and generate() settings.

A language model does not write a sentence in one go. At every step it outputs a probability distribution over the whole vocabulary for the next token. A decoding strategy decides how to pick a token from that distribution, append it to the text and repeat. The same model can sound careful, creative or repetitive depending only on these settings.

The generation loop

flowchart LR
  A["Prompt tokens"] --> B["Model: logits for next token"]
  B --> C["Decoding rule picks a token"]
  C --> D["Append to sequence"]
  D --> E{"EOS or max_new_tokens?"}
  E -->|"no"| B
  E -->|"yes"| F["Decode to text"]

Every call to generate() runs this loop; only the "decoding rule" box changes.

Real code (read-only):

Python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tok("The future of open models is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=30, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Prefer max_new_tokens (tokens to add) over max_length (prompt plus output).

Always pick the most probable token. It is fast and deterministic, but it can fall into repetitive loops and may miss better overall sentences because a locally best word can lead to a poor continuation.

Keep the num_beams best partial sequences at each step, extend each of them, and keep the best again by total log-probability. It explores more of the space and is the usual choice for translation and summarization, where one good answer exists. It is poor for open-ended chat because it prefers bland, high-probability text.

Sampling, temperature, top-k, top-p

Sampling draws the next token randomly according to the probabilities, so outputs vary between runs. Three controls shape that randomness:

  • Temperature divides the logits before softmax. Below 1 the distribution gets sharper (safer text); above 1 it gets flatter (more surprising). Temperature 0 is greedy.
  • Top-k keeps only the k most likely tokens and renormalises.
  • Top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to p, so the number of candidates adapts to how confident the model is.

The runnable code below is a stand-in: it applies each rule to a hand-written next-token distribution so you can see how the candidate sets change. It is not a language model.

Python Stand-in: temperature, top-k and top-p

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Low temperature concentrates probability on "the"; high temperature spreads it so that even unlikely words such as "banana" get a chance. Top-k and top-p cut off the long tail of absurd candidates ("quantum") before sampling.

In generate():

Python
out = model.generate(
    **inputs, max_new_tokens=60,
    do_sample=True, temperature=0.7, top_p=0.9, top_k=50,
)

Setting do_sample=True is required for temperature, top-k and top-p to have any effect.

Beam search by hand

The stand-in below runs beam search over a tiny hand-made "model" (a table of next-word probabilities) to show why it can beat greedy search.

Python Stand-in: greedy versus beam search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy takes "a" first (0.55 beats 0.45) but the best complete sentence is "the plan" (0.45 x 0.95 = 0.43 versus 0.55 x 0.5 = 0.275). Beam search keeps both starts alive and finds it.

Other useful settings

Setting Effect
repetition_penalty (e.g. 1.1) lowers the score of tokens already used
no_repeat_ngram_size=3 forbids repeating any 3-gram
num_return_sequences returns several candidates
eos_token_id, stop_strings decide when to stop
min_new_tokens prevents answers that are too short
seed via torch.manual_seed reproducible sampling

A model's recommended defaults live in generation_config.json; read them before overriding.

Streaming

For chat interfaces, TextStreamer (or TextIteratorStreamer in a thread) prints tokens as they are produced, so users see text appear instantly instead of waiting for the whole answer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does lowering the temperature do?

    Logits are divided by the temperature, so values below 1 emphasise the top tokens.
  2. 2. What does top-p = 0.9 keep?

    Nucleus sampling adapts the number of candidates to the model's confidence.
  3. 3. Which task usually benefits most from beam search?

    Translation has a narrow set of good outputs, and beam search finds high-probability ones.
  4. 4. temperature=0.7 has no effect unless:

    Without sampling, generate() picks the argmax and ignores temperature, top-k and top-p.

Summary

  • Generation repeats: logits, decoding rule, append, until EOS or the token limit.
  • Greedy and beam search are deterministic; sampling adds randomness controlled by temperature, top-k and top-p.
  • Beam search finds better whole sequences than greedy for constrained tasks.
  • Use max_new_tokens, repetition controls and stopping rules; stream output for interactive apps.

Next: Batching, Devices and Speed