Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 53 ยท Getting Started

Calling an LLM API

Learn how LLM APIs work: messages and roles, system prompts, sampling parameters, streaming, error handling with retries, and keeping API keys safe.

Using a hosted language model means sending an HTTP request and reading the response. The details differ between vendors, but nearly all chat APIs share the same shape: a list of messages with roles, a handful of parameters, and a response that holds the generated message plus usage numbers. Once you understand that shape, switching providers is mostly renaming fields.

The site cannot call a real model from your browser, so the request code below is read-only. To still let you experiment, the live examples use a fake client that returns responses in the same structure, which is also how you test your own code without paying for calls.

The shape of a chat request

A request names a model and carries the conversation so far. The model itself is stateless: it remembers nothing between calls, so you send the history every time.

Python
# READ-ONLY: generic sketch. Field names vary slightly between vendors.
import os
import requests

response = requests.post(
    "https://api.example-llm.test/v1/chat",
    headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"},
    json={
        "model": "some-model-name",
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "Explain recursion in one sentence."},
        ],
        "temperature": 0.2,
        "max_tokens": 100,
    },
    timeout=30,
)
data = response.json()
print(data["message"]["content"])

Each message has a role:

  • system: instructions about behaviour, tone and rules. It is written by the developer and applies to the whole conversation.
  • user: what the person (or your program) asks.
  • assistant: earlier model replies. You include them so the model sees the conversation history.
  • tool: results of tool calls, covered in Tool Use and Function Calling.

Anatomy of a request

Model Which model answers.
Messages System, user and assistant turns.
Parameters Temperature, max tokens, stop.
API key Proves who is paying.
1 / 2

Parameters you will use

  • temperature: randomness of word choice. Low values give focused, repeatable answers; high values give variety (see Logits, Softmax and Temperature).
  • top_p (and sometimes top_k): limit choices to the most likely tokens (Top-k, Top-p and Min-p Sampling). Change temperature or top_p, not both at once, unless you have a reason.
  • max_tokens: the largest number of output tokens. It protects your budget and latency. If the answer reaches it, the text is cut off and the finish reason says so.
  • stop sequences: strings that end generation when produced.
  • seed: some APIs accept a seed to make sampling more repeatable, though perfect determinism is not guaranteed.

System prompts

The system message is the best place for stable rules: role, tone, output format and what to refuse. Keep the user message for the changing request. System text is not secret or magically safe: a user can try to override it, which is why Prompt Injection and LLM Security matters. The craft of writing both is in The Anatomy of a Good Prompt.

A fake client with the real shape

The class below mimics a chat API. It is deterministic, so you can write and test the code around a model before touching a real one.

Python A fake LLM client

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice finish_reason becomes length when max_tokens cuts the answer. Always check it: a silently truncated JSON answer is a classic bug. Token counts in usage are what you are billed on (Counting Tokens and Estimating Cost).

Streaming

A long answer can take many seconds. With streaming the server sends tokens as they are produced (usually as server-sent events), so your interface shows text immediately and users feel the app is fast. You read small chunks, each holding a piece of text, and join them.

Python
# READ-ONLY: typical streaming loop with a vendor SDK (names vary).
stream = client.chat_stream(model="some-model-name", messages=messages)
for chunk in stream:
    piece = chunk["delta"]
    print(piece, end="", flush=True)

The last chunk usually carries the finish reason and token usage. If the connection drops halfway you will have partial text, so decide whether to retry or show what you have.

Python Consuming a stream of chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Errors and retries

Network calls fail. Learn the common HTTP status codes:

Status Meaning What to do
400 Bad request (invalid field, prompt too long) Fix the request; do not retry unchanged
401 / 403 Bad key or no permission Check the key; do not retry
429 Rate limit or quota Wait, then retry with backoff
500 / 503 Server problem or overload Retry with backoff
Timeout No answer in time Retry, maybe with a smaller request

Exponential backoff waits a little longer after each failure (1 s, 2 s, 4 s...), usually with a bit of random jitter so many clients do not retry at the same moment. Retry only errors that can succeed later, and cap the number of attempts.

Python Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The example records the delays instead of sleeping, so it runs instantly. Production versions of this idea, with rate limits and fallbacks, appear in Deploying and Monitoring LLM Apps.

API keys and secrets

An API key is a password that spends money.

  • Never put a key in front-end JavaScript, a public repository or a screenshot. Call the model from your server, and let the browser call your server.
  • Store keys in environment variables or a secrets manager, as in os.environ["LLM_API_KEY"] above.
  • Use separate keys for development and production, set spending limits, and rotate a key at once if it leaks.
  • Be careful what you send: prompts may contain personal or confidential data. Read the vendor's data-retention terms.

If a key is ever committed to Git, deleting the file is not enough, because it remains in history. Revoke the key and create a new one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must you resend earlier messages with each API call?

    Chat APIs are stateless. Your program stores the conversation and sends the relevant history every time.
  2. 2. A response has finish_reason equal to length. What does it mean?

    A length finish reason means generation was stopped by the token limit, so the text can be incomplete, for example truncated JSON.
  3. 3. Which error should NOT be retried unchanged?

    A 400 means the request itself is wrong, so sending the same thing again will fail again. Rate limits and server errors can succeed later.
  4. 4. Where should an LLM API key live?

    Keys are secrets. Keep them on the server and let clients talk to your server, never directly to the vendor with your key.

Summary

  • Chat APIs take a model name, a list of role-tagged messages and parameters, and return a message, a finish reason and token usage.
  • The API is stateless: you resend the conversation each time.
  • Temperature and top_p shape randomness; max_tokens and stop sequences limit length; check finish_reason for truncation.
  • Streaming delivers chunks as they are generated, improving perceived speed.
  • Retry 429 and 5xx errors with exponential backoff and jitter; do not retry 400-level mistakes.
  • Keep API keys on the server, out of Git, with spending limits and rotation.

Next: look inside the text the model actually sees in Tokens and Tokenization.