Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 63 ยท Neural NLP and Transformers

Large Language Models and Prompting

See how large language models generate text token by token, how temperature, top-k and top-p sampling work, and which prompt patterns make answers reliable.

A large language model (LLM) is a decoder-only Transformer with billions of parameters, pre-trained on a vast amount of text and then usually tuned to follow instructions and chat. From the outside it looks like a program you talk to. On the inside it does one thing, over and over: given the tokens so far, it produces a probability for every possible next token.

Understanding this loop explains almost everything about LLM behavior: why the same prompt can give different answers, why wording matters, why long inputs cost money, and why a model can state a falsehood with total confidence. In this lesson you will build the generation loop, implement the sampling settings you see in every API, and learn prompt patterns that work.

How an LLM generates text

The generation loop

Tokenize Split the prompt into subword tokens.
Score The model rates every token in its vocabulary.
Pick Choose one token using a sampling rule.
Repeat Append it and run again until a stop token.
1 / 2

Text is first cut into subword tokens. A common English word is usually one token, rare words are several pieces, and a rough rule of thumb is that a token is about four characters, or three quarters of a word. The model turns the token list into a vector of logits, one raw score per vocabulary token. A softmax converts the logits into probabilities. Then a sampling rule picks one token, the token is appended to the input, and the whole process repeats. The model has no plan for the full answer; each word is chosen given only what came before.

Sampling: temperature, top-k and top-p

If you always pick the highest-probability token (greedy decoding), output is repeatable but often dull and prone to loops. Sampling from the distribution adds variety. Three settings control it:

  • Temperature divides the logits before the softmax. Below 1 the distribution gets sharper (the favorite wins more often); above 1 it flattens (unlikely tokens get a chance); a value near 0 approaches greedy decoding.
  • Top-k keeps only the k most likely tokens and renormalizes.
  • Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least p. Unlike top-k, the set size adapts: it is small when the model is confident and larger when it is unsure.

The code applies all three to the same made-up scores for the word after "The cat sat on the".

Python Temperature, top-k and top-p

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At temperature 0.5, mat takes 76% of the probability; at 1.0 it has 53%; at 2.0 only 36% and even banana gets 3%. Top-k with k = 3 removes the three unlikely words and scales the rest up (mat becomes 0.56). Top-p with p = 0.7 keeps just two words, because mat (0.53) plus sofa (0.24) already pass 0.7; had the model been more confident, the set would have shrunk to a single word, and if it were unsure the set would have grown. The last line draws 1000 samples at temperature 2: mat appears about 36% of the time, matching its probability. In practice, use low temperature (0 to 0.3) for extraction, code and factual answers, and higher values (0.7 to 1.0) for brainstorming and creative text. Most APIs let you set temperature and either top-p or top-k.

A tiny model, to see temperature at work

Real LLMs are far too large to run here, but the sampling loop is identical for a toy model. The next example builds a bigram model (the idea from n-gram language models) on a few sentences and generates text with a seeded random generator, so the output is repeatable.

Python Generating text with different temperatures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every run is a path through a probability table, and the seed decides which path. With the same temperature, different seeds give different sentences; that is why the same prompt can give different answers. Our corpus is tiny, so the effect of temperature is mild here: at 0.2 the model keeps taking the most common next word, while at 3.0 it more often picks rare continuations such as a mat right after a full stop. A real model with a vocabulary of 100,000 tokens shows a much stronger difference, from repetitive and safe to wild and incoherent.

Prompt patterns that work

Because the model continues text, the prompt is the only way you steer it. A few patterns cover most needs.

  • Be specific about the task and the format. "Summarize in three bullet points, each under 15 words" beats "summarize".
  • Give a role and context. "You are a support agent for a bike shop" narrows tone and vocabulary.
  • Few-shot examples. Show two or three input-output pairs in the prompt. The model picks up the pattern, including the output format, without any training. A prompt with no examples is zero-shot.
  • Ask for step-by-step reasoning on multi-step problems ("think through this step by step, then give the answer"). Writing out intermediate steps often improves accuracy on math and logic.
  • Separate instructions from data with delimiters, and tell the model that the text between them is data, not commands.
  • Ask for structured output (JSON with named fields) when code will read the answer, then validate it.

The program assembles a few-shot prompt with an f-string. Real prompts are just strings built in code like this.

Python Building a few-shot prompt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prompt ends exactly where the answer should begin (Sentiment:), so the most likely continuation is a label in the same style as the examples. The final line estimates the cost: 255 characters is about 63 tokens, and you pay for those on every call, which is a good reason to keep examples short. Never put secrets in prompts, and keep in mind that a prompt that contains user text can be hijacked: if that text says "ignore the above instructions", a model may obey. This is called prompt injection, and the defense is to treat the model's output as untrusted, limit what tools it can use and validate everything.

Hallucinations and limits

An LLM is trained to produce plausible text, not true text. When it lacks a fact, it often still produces something fluent, a hallucination: an invented citation, a wrong date, a function that does not exist. The causes are built in: the model's knowledge stops at its training date, it compresses billions of documents into its weights and cannot look things up, and a confident tone costs it nothing.

Ways to reduce the problem:

  • Ground the answer in documents you provide, the idea of retrieval and RAG, and tell the model to answer only from them or say it does not know.
  • Ask for sources and verify them; ask for quotes from the supplied text.
  • Lower the temperature for factual tasks, and check outputs with code where possible (run the code, validate the JSON, compute the number).
  • Keep humans in the loop for anything high-stakes.

Other limits worth knowing: the context window is finite, models are weak at exact arithmetic and counting letters (tokens hide the letters), output can vary between runs, and quality depends on the language, since less-represented languages get worse results and use more tokens.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does an LLM compute at each step of generation?

    The model scores every token in its vocabulary given the text so far, a sampling rule picks one, and the loop repeats until a stop token.
  2. 2. What happens when you raise the temperature from 0.5 to 2.0?

    Dividing the logits by a larger number brings the probabilities closer together, which makes output more varied and less predictable.
  3. 3. How does top-p sampling differ from top-k?

    Top-k uses a fixed number of candidates, while top-p lets the number of candidates grow or shrink with the model's confidence.
  4. 4. Which prompt technique shows the model the desired pattern with a few input-output pairs?

    In few-shot prompting you place a handful of worked examples in the prompt, and the model continues the same pattern without any weight update.
  5. 5. Which measure best reduces hallucinations in a question-answering app?

    Grounding the answer in supplied text gives the model facts to copy from instead of relying on memory. Verification and low temperature help further.

Summary

  • An LLM repeats one loop: tokenize, score every next token, pick one, append, repeat.
  • Temperature reshapes the probabilities; top-k and top-p cut off unlikely tokens; seeds and sampling make answers vary.
  • Good prompts state the task and format, give context, show examples, separate data from instructions and ask for structured output.
  • Prompts cost money per token, and user text in a prompt can attempt prompt injection.
  • Hallucinations are plausible text without a source of truth; ground answers in documents, verify outputs and keep humans involved for high stakes.

Next, learn how to give a model the right documents in Retrieval and RAG.