Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 38 ยท Getting Started

How Language Models Read Prompts

Understand tokens, context windows and next-token prediction, the mental model that explains why prompts behave the way they do. Includes runnable Python.

You do not need to know how to train a model to write good prompts, but you do need a working mental model of what happens to your text. Three ideas cover most of it: text becomes tokens, the model sees a limited context window, and it generates output by repeatedly predicting the next token.

Tokens: the units a model actually sees

A model does not read letters or whole words. A tokenizer splits text into chunks called tokens, each mapped to a number. Common words are often one token; rare words, names, and code are split into several. Spaces, punctuation and capitalization all affect the split.

Real tokenizers (byte-pair encoding and similar) learn their vocabulary from data. The block below is a toy stand-in: it splits on words and punctuation, then breaks long words into pieces of at most four characters. Real counts differ by model, but the behavior (long and unusual words cost more) is realistic.

Python Toy tokenizer (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why this matters to you:

  • Cost and speed are usually measured in tokens, for both input and output.
  • Limits (context window, maximum output) are in tokens.
  • Oddities such as trouble counting letters in a word or reversing a string come from the model seeing tokens, not characters.

A rough rule of thumb for English prose is that one token is about three quarters of a word, but treat that as an estimate and use your provider's tokenizer for real budgets.

The context window

The context window is the maximum number of tokens a model can consider at once. It holds everything: the system prompt, the conversation so far, any retrieved documents, your question, and the answer being generated. When the total would exceed the window, something must be cut or the request fails.

flowchart TB
  subgraph W["Context window (fixed token budget)"]
    S["System prompt"]
    H["Conversation history"]
    D["Documents / examples"]
    Q["User question"]
    A["Generated answer"]
  end

Everything shares one budget. A long document leaves less room for history and for the answer.

That has practical consequences. If you paste a 50-page document and ask for a detailed answer, you may leave no space for the answer. If a chat runs long, early messages may be dropped or summarized. And even when text fits, models can attend less reliably to material buried in the middle of a very long prompt, so placement matters (more on that in the retrieval chapter).

Python Budgeting a context window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run it, then shrink history or document until the budget check passes. That trade-off, what to keep and what to drop, is a daily decision in context engineering, which gets its own chapter later.

Next-token prediction

At its core, a language model reads the tokens so far and produces a probability distribution over what token could come next. It picks one (how it picks is controlled by settings you will meet in the next lesson), appends it, and repeats until it emits a stop signal or hits a length limit.

This explains a lot of prompt behavior:

  • The model continues the pattern it sees. If your prompt looks like a formal report, you get a formal report. If it ends mid-sentence, it may finish the sentence.
  • Early tokens influence later ones. A wrong or sloppy start to an answer tends to carry forward, which is why asking for a plan first can improve results.
  • Examples in the prompt are very strong signals, because they are the nearest patterns to copy.

The next block is a stand-in that predicts the next word from a hand-written table, so you can watch generation happen one step at a time. It is deliberately tiny and not a real model.

Python Generating one token at a time (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A real model has a vocabulary of tens of thousands of tokens and conditions on the whole prompt, not just the last word, but the loop (predict, append, repeat) is the same.

What models do not do

Keep three limits in mind as you read the rest of the course:

  1. They have no memory between requests unless you resend the earlier text. A "conversation" is the application re-sending the history each turn.
  2. They do not look things up unless you give them a tool or paste the information into the prompt. Their built-in knowledge stops at a training cutoff and can be wrong.
  3. They produce plausible text, not guaranteed facts. A confident tone is not evidence. This is why validation and evaluation (chapter 6) matter.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a token?

    Tokenizers split text into chunks. Common words may be one token while rare words are split into several.
  2. 2. Which of these counts against the context window?

    The window is one shared budget for all input and output tokens.
  3. 3. Why might a model be bad at counting the letters in a word?

    Because words are split into tokens, character-level operations are not directly visible to the model.
  4. 4. How does a model generate a long answer?

    Generation is a loop of predicting a token, appending it, and predicting again.

Summary

  • Text becomes tokens; cost, speed and limits are measured in tokens.
  • The context window is one shared budget for system prompt, history, documents, question and answer.
  • Models generate by predicting the next token repeatedly, so they continue the patterns your prompt sets up.
  • Models are stateless, do not look things up on their own, and can sound confident while being wrong.

Next: Model Settings: Temperature and Tokens