Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 38 ยท Getting Started

Model Settings: Temperature, Top-p and Max Tokens

Learn what temperature, top-p, max tokens and stop sequences do to a model's output, with numpy simulations that show how sampling really works.

Besides the prompt, you usually control a handful of generation settings whenever you call a model. They do not change what the model knows; they change how it chooses among the possible next tokens and when it stops. Knowing them helps you diagnose problems that look like prompt bugs but are really settings.

Exact names and ranges vary by provider (and some newer models fix or hide a few of them), so check your provider's documentation. The ideas below are general.

From scores to probabilities

At each step the model produces a score (a logit) for every token in its vocabulary. A function called softmax turns scores into probabilities that add up to 1. Then the decoder picks a token. The settings below adjust that last step.

Temperature

Temperature rescales the scores before softmax: each score is divided by the temperature.

  • Low temperature (near 0) sharpens the distribution. The most likely token dominates, so output is focused and repeatable.
  • Temperature 1 leaves the distribution as the model produced it.
  • High temperature flattens the distribution. Unlikely tokens get a real chance, so output is more varied and more error-prone.

The simulation below uses made-up scores for five candidate words after "The weather today is". It is an illustration of the math, not a real model.

Python How temperature reshapes probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 0.2 almost all the probability sits on "sunny". At 2.0 even "purple" has a noticeable chance. A temperature of exactly 0 is usually treated as greedy decoding: always take the top token.

Choosing a temperature

Task Typical choice
Extraction, classification, code, math Low (0 to 0.3)
Q&A over documents Low to medium
Brainstorming, naming, creative writing Medium to high

Even at temperature 0 some providers are not perfectly deterministic (hardware and batching can change results slightly), so never assume byte-identical output.

Top-p (nucleus sampling) and top-k

Instead of reshaping the whole distribution, these settings cut off the tail.

  • Top-p keeps the smallest set of tokens whose probabilities add up to p (for example 0.9), then samples among them.
  • Top-k keeps only the k most likely tokens.

Providers generally advise adjusting temperature or top-p, not both at once, because their effects overlap.

Python Top-p keeps the likely core

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With top_p=0.7, only "sunny" and "cloudy" survive, so odd words can never appear.

Sampling with a fixed seed

Because sampling is random, two runs can differ. Some providers offer a seed to make sampling repeatable. The block below samples from the same distribution with and without a seed, using numpy.

Python Random sampling and seeds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Runs A and B match because they share a seed; the third differs.

Maximum tokens and stop sequences

  • Max tokens (sometimes max_output_tokens) caps the length of the answer. If the model reaches the cap mid-sentence, the answer is simply cut off. A truncated JSON object is a common cause of parse errors, so leave headroom when you ask for structured output.
  • Stop sequences are strings that end generation when produced, such as "\n\n" or "</answer>". They are handy when you want exactly one item, or one tagged section.
  • Frequency and presence penalties (where offered) discourage repetition by lowering the score of tokens that already appeared.

The next block mimics a stop sequence and a length cap on a pre-written string, the way an API trims output for you.

Python Stop sequences and length caps (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Debugging checklist

Symptom Setting to check
Answer ends mid-sentence Max tokens too low
Output changes every run Temperature or top-p too high
Output too repetitive or dull Temperature too low, penalties
Extra chatter after the answer Add a stop sequence or tighten the prompt
Nonsense words appear Temperature too high

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does lowering the temperature do?

    Dividing scores by a small temperature sharpens the distribution toward the top token.
  2. 2. What does top-p = 0.9 mean?

    Nucleus sampling keeps the smallest set of tokens covering probability p and discards the unlikely tail.
  3. 3. A JSON reply is cut off and fails to parse. What is a likely cause?

    When the output length cap is hit, the answer stops wherever it is, leaving invalid JSON.
  4. 4. For a classification task you want the same label every time. What is a sensible setting?

    Low temperature favors the most likely answer, which gives consistent labels.

Summary

  • Settings control how the next token is chosen and when generation stops; they do not change what the model knows.
  • Low temperature gives focused, repeatable output; high temperature gives variety and more mistakes.
  • Top-p and top-k cut off the unlikely tail; adjust temperature or top-p, not both.
  • Max tokens can truncate answers; stop sequences end them early. Leave headroom for structured output.

Next: The Anatomy of a Prompt