Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 41 ยท Serving and Monitoring

Batch and Online Model Serving

Choose between batch, online (real-time) and streaming inference. Compare latency, cost and freshness, and see working examples of batch scoring and a cached online predictor.

A trained model creates value only when its predictions reach a person or a system. Serving is everything between "a model file exists" and "predictions are being used". There are three common patterns, and choosing the right one early saves a lot of money and pain.

  • Batch inference: score many rows at once on a schedule (nightly churn scores, weekly demand forecasts). Results are stored and looked up later.
  • Online inference: score one request at a time while a user waits (fraud check at payment, search ranking). Answers must arrive in milliseconds.
  • Streaming inference: score events continuously as they arrive on a stream, usually with a short delay (sensor alarms, clickstream).

Three serving modes

Batch Many rows, on a schedule, stored results.
Online One request, answer in milliseconds.
Streaming Continuous events, near real time.

Freshness costs money. Pick the slowest mode that works.

1 / 2

How to choose

Ask three questions:

  1. Does the prediction depend on information that only exists at request time? (the item in the cart right now, the transaction being approved). If yes, you need online inference.
  2. How stale can a prediction be? If a score that is a day old is fine, batch is almost always simpler and cheaper.
  3. How many different inputs can there be? If the set of entities is limited (all 2 million customers), you can precompute a score for each and look it up instantly, which gives online speed with batch simplicity.
flowchart TD
  Q1{"Needs data from the request moment?"} -->|yes| ON["Online inference"]
  Q1 -->|no| Q2{"Can the score be hours old?"}
  Q2 -->|yes| BA["Batch inference + lookup table"]
  Q2 -->|no| Q3{"Events arrive as a stream?"}
  Q3 -->|yes| ST["Streaming inference"]
  Q3 -->|no| ON

A rough decision path. Real systems often combine modes.

Batch inference in practice

A batch job reads a table, loads the model once, scores everything in chunks that fit in memory and writes results with enough metadata to trace them: model version, score time and the input key.

Python Chunked batch scoring

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is a table keyed by customer_id. The application does not call the model; it reads a score from a database. That is cheap, scales trivially and can be inspected, and it also gives you a record of what the model said for later audits.

Write batch output to a new, versioned location and switch a pointer when the whole job succeeds. Then a half-finished job never leaves users with a mix of old and new scores.

Online inference in practice

An online service keeps the model in memory, validates the incoming request, builds features, predicts and returns, all in a few milliseconds. Costs to watch:

  • Cold start: loading the model takes long, so load it once at startup, never per request.
  • Feature lookup: the request often carries only an ID, and the service must fetch the rest (from a feature store or cache) quickly.
  • Repeated requests: the same inputs are often asked again, so a cache helps.
Python An online predictor with a cache

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rounding the features before caching is a design choice: it raises the cache hit rate at the cost of tiny differences in input. Decide that with the model owner, because for some models it changes results unacceptably.

Streaming and micro-batches

For events on a stream (Kafka, Kinesis, Pub/Sub), you can score each event as it arrives or group events into micro-batches every few hundred milliseconds. Micro-batching trades a little latency for much better hardware use, because many models predict a block of rows barely slower than one row.

Python Why micro-batching helps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exact numbers depend on the machine, so do not memorize them; the direction (batched is much faster per row) is stable.

Hybrid designs

Real systems mix modes:

  • Precomputed scores with online refinement: batch gives a base score; a light online model adjusts it using the current session.
  • Two-stage ranking: a cheap model narrows millions of items to a few hundred, then a heavier online model ranks those.
  • Fallbacks: if the online model is down or too slow, return the last batch score or a safe default rather than an error.

Common mistakes

  • Choosing online "because it feels modern". It costs more and is harder to operate. Use batch when staleness is acceptable.
  • Loading the model on every request. Load once at startup.
  • No model version in the output. You will need it to debug.
  • Different features in batch and online. Share the same feature code (training-serving skew).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A nightly job scores every customer for churn risk and stores the results. Which mode is this?

    Many rows on a schedule with stored results is batch inference.
  2. 2. A payment must be approved or declined while the customer waits, using the current transaction amount. Which mode fits?

    The decision depends on data that exists only at request time and must be fast.
  3. 3. Where should a model be loaded in an online service?

    Loading is slow; do it once and keep the model in memory.
  4. 4. What is a benefit of micro-batching?

    You accept a small delay to gain throughput.

Summary

  • Batch, online and streaming serve different freshness and cost needs; pick the simplest that works.
  • Batch results are cheap, traceable and can be served by lookup.
  • Online services load the model once, validate input, fetch features fast and often cache.
  • Micro-batching increases throughput at a small latency cost.
  • Hybrids and fallbacks are common in production.

Next: Building a Prediction Service