Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 32 ยท Models, Prompts and Parsers

Chat Models in Depth

Work with LangChain chat models in depth: init_chat_model, temperature and other parameters, streaming, token usage, timeouts, retries and switching providers safely.

A chat model is the LangChain wrapper around a provider's chat API. Every one of them (ChatOpenAI, ChatAnthropic, ChatGoogleGenerativeAI, ChatOllama ...) is a Runnable with the same methods, so choosing a model is a configuration decision rather than an architecture decision. This lesson covers how to create one, the parameters that matter, and the production habits around them.

Creating a model

There are two equivalent ways. The universal helper picks the class from a provider:model string:

Python
from langchain.chat_models import init_chat_model

model = init_chat_model("anthropic:claude-sonnet-4-5", temperature=0)

Or import the provider class directly. This is more explicit and gives access to provider-only options:

Python
from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini", temperature=0, timeout=30, max_retries=2)

Use the helper when you want to switch providers from a config file, and the direct class when you need something only that provider offers. Model names change often; check your provider's current list rather than copying names from old articles.

Parameters worth knowing

Parameter Meaning Typical use
temperature Randomness of sampling. 0 is as deterministic as the provider allows; higher is more varied 0 for extraction and classification, 0.7 and up for creative writing
max_tokens Upper limit on reply length Control cost and prevent runaway output
timeout Seconds to wait for a reply Always set in production
max_retries Automatic retries on transient errors (rate limits, network) 2 to 5
top_p, stop, seed Sampling cutoff, stop sequences, best-effort reproducibility Provider-dependent
api_key, base_url Credentials and a custom endpoint (proxies, local servers) Usually left to environment variables

Some newer "reasoning" models ignore or reject temperature; the provider's docs say which options each model accepts. Providers also accept their own extras (for example reasoning effort); those are passed as keyword arguments to the provider class.

Temperature 0 does not make a model perfectly repeatable, only more consistent. Never write a test that expects an identical string from a live model; check properties of the answer instead.

Calling, streaming and batching

Python
reply = model.invoke("Name three sorting algorithms.")
print(reply.content)

for chunk in model.stream("Explain hashing in two sentences."):
    print(chunk.content, end="", flush=True)

answers = model.batch(["Capital of France?", "Capital of Japan?"])

Streaming yields AIMessageChunk objects. Chunks can be added together with +, which is how a UI collects the final message while still displaying partial text.

Token usage and cost

Providers bill by tokens, so you should know how many you use. The reply reports it:

Python
reply = model.invoke("Summarize TCP in one line.")
usage = reply.usage_metadata        # {'input_tokens': ..., 'output_tokens': ..., 'total_tokens': ...}

To see the same totals across several calls, LangChain offers callbacks that accumulate usage. A rough cost is input_tokens * input_price + output_tokens * output_price, using your provider's per-token prices. Long chat histories and large retrieved passages are the usual source of surprise bills, which is why this course returns to trimming context in the memory chapter.

A tracking wrapper (runnable)

This stand-in imitates the pattern: a fake model that reports token counts (roughly one token per word), and a wrapper that accumulates totals and estimates cost.

Python Stand-in: token usage and cost tracking

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Retries, timeouts and rate limits

Provider APIs fail sometimes: a 429 (rate limit), a 500, a dropped connection. Built-in max_retries handles short blips with exponential backoff. For heavier loads, LangChain provides a rate limiter you attach to the model so a batch of 500 inputs does not trigger a storm of 429s:

Python
from langchain_core.rate_limiters import InMemoryRateLimiter

limiter = InMemoryRateLimiter(requests_per_second=2)
model = ChatOpenAI(model="gpt-4o-mini", rate_limiter=limiter)

Here is the backoff idea in plain Python. The fake API fails twice, then succeeds.

Python Stand-in: retry with exponential backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Local and open models

You do not need a cloud account to learn. A local runtime such as Ollama serves open-weight models on your machine, and langchain-ollama provides ChatOllama with the same interface. A response from a small local model is usually weaker, but the code is identical, which is the whole point of the abstraction. base_url also lets you point ChatOpenAI at any OpenAI-compatible server.

Choosing a model

Start from the task, not the brand:

  • Extraction, classification, routing: small, cheap, fast model at temperature 0.
  • Writing and reasoning: larger model; consider a reasoning model for hard problems.
  • Long documents: check the context window, and still retrieve rather than paste everything (see retrieval).
  • Privacy constraints: local or self-hosted model.

Measure with your own examples. A cheaper model that passes your tests beats an expensive one you never compared.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does init_chat_model('anthropic:some-model') do?

    It chooses the right provider class from the prefix and returns a normal chat model Runnable.
  2. 2. For classifying support tickets into fixed categories, which temperature is usually best?

    Low temperature gives the most consistent, least creative output, which suits classification.
  3. 3. Where do you find the tokens used by one call?

    usage_metadata reports input, output and total tokens for providers that expose them.
  4. 4. Why attach a rate limiter when running batch() on hundreds of inputs?

    batch sends many requests at once; a limiter spaces them so the provider does not reject them.

Summary

  • Create chat models with init_chat_model or a provider class; both give a Runnable.
  • Key parameters: temperature, max_tokens, timeout, max_retries; check each model for what it accepts.
  • Replies expose token usage; track it to control cost.
  • Use retries, timeouts and rate limiters for reliability; local models use the same interface.
  • Pick the model per task and test with your own examples.

Next: Prompt Templates