Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 39 ยท Getting Started

LLMs as the Agent's Brain

How a language model fits inside an agent: stateless calls, roles and messages, tokens, context windows, temperature and the limits you must design around.

At the center of most modern agents sits a large language model. From the program's point of view it is a function: you pass in a list of messages and get back new text (and sometimes a request to call a tool). That is all. Everything that makes an agent feel intelligent, such as memory, planning and tool use, is built in the code around that function. Understanding exactly what the function does, and what it does not do, saves you from many early mistakes.

A model call is stateless

When you chat with an assistant it feels like the model remembers you. It does not. Each call receives the whole conversation so far and produces the next reply. If your program forgets to send an earlier message, the model has no idea it existed.

A typical request, shown read-only because it needs a key and network, looks like this:

Python
# Read-only: shape of a real chat request (provider SDKs differ in details)
response = client.messages.create(
    model="a-model-name",
    max_tokens=500,
    system="You are a careful research assistant.",
    messages=[
        {"role": "user", "content": "What is the capital of Australia?"},
    ],
)
print(response.content[0].text)

Three parts matter for agents:

  • System prompt: standing instructions (role, rules, format). It sits above the conversation.
  • Messages: alternating user and assistant turns. Tool results also arrive as messages, as you will see in the tools chapter.
  • Parameters: the model name, the maximum length of the reply, and sampling settings such as temperature.

Building the message list yourself

The agent program owns the list. Each turn it appends the new user input, calls the model with the full list, then appends the reply. The stand-in below mimics a model that only knows what is in the list it receives, so you can watch statelessness happen.

Python Stand-in: the model only sees what you send

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same question gets two different answers only because of what the program sent. This is why agent state is a first-class design topic.

Tokens and the context window

Models read and write tokens: chunks of text, roughly three-quarters of an English word on average. Every model has a context window, the maximum number of tokens that the system prompt, the whole message list and the reply can occupy together. A long agent run produces many tool results, and they all land in that list, so the window fills up. When it does, the call fails or you must shorten the history.

Costs scale with tokens too, usually priced per million input tokens and per million output tokens, with output costing more. A loop that resends a growing history at every step uses far more input than the final answer suggests.

Python Estimate tokens and trim history to fit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dropping the oldest messages is the crudest strategy: you can lose a fact the agent still needs. Better options, such as summarizing and retrieval, are covered in the state lesson and the memory chapter.

Sampling and determinism

At each position the model produces probabilities for the next token and then samples. Temperature controls how adventurous the sampling is: near 0 the model picks the most likely token almost every time, higher values spread choices out. For agents that must call tools with exact arguments, low temperature is usually the safer default. Even at zero, outputs are not guaranteed to be identical between runs, so never write code that assumes a precise string comes back.

What the model cannot do alone

Keep this list in mind when designing an agent:

  • It cannot see anything outside the messages you send, including files, the web or the time.
  • It cannot act: it only emits text. Your program decides whether to execute what it asks for.
  • It can be confidently wrong, so important claims need verification by a tool or a check.
  • It can be steered by text inside its inputs, including text from a web page or a file (prompt injection, covered in the safety chapter).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an LLM API call receive that makes the model appear to remember the conversation?

    Calls are stateless. Memory appears because the program resends the conversation each time.
  2. 2. What happens when an agent's accumulated messages exceed the context window?

    The window is a hard limit on prompt plus reply. You must trim, summarize or retrieve selectively.
  3. 3. For an agent that must produce exact tool arguments, which temperature is a sensible default?

    Low temperature makes the most likely token win most of the time, which reduces random variation in arguments.
  4. 4. Why does a long agent loop cost more than the final answer suggests?

    Input tokens accumulate: step N pays for everything from steps 1 to N-1 again.

Summary

  • A model call is a stateless function from messages to a reply.
  • Your program owns the message list, the system prompt and the loop.
  • Tokens bound both cost and the context window; long runs must manage their history.
  • Low temperature favors consistent tool arguments, but never assume exact outputs.
  • The model only emits text: your code decides what actually happens.

Next: Your First Agent Loop