Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 73 ยท Frameworks and Production

Deploying and Serving Models

Take a trained network to production: save weights, export with ONNX, shrink it with int8 quantization and pruning, serve it, and watch for data drift.

A trained network that lives only in a notebook is a demo. To be useful it has to run somewhere else: on a server answering web requests, in a nightly batch job, on a phone with a small battery. That move is deployment, and it brings new questions. How do you store the weights safely? How do you run the model without your whole training stack? How do you make it small and fast enough? And how do you notice when it quietly stops working because the world changed?

This lesson covers the path from trained weights to a monitored service. Two techniques that make models smaller, quantization and pruning, are demonstrated with runnable NumPy code on a real (small) digit classifier. Production tooling such as ONNX and model servers is shown as read-only snippets.

From notebook to service

Save Weights plus the config that builds the model.
Export A portable format, not your training code.
Shrink Quantize or prune to save memory and time.
Serve Batch jobs or an always-on endpoint.

Then watch it, because production data keeps changing.

1 / 2

Saving and loading weights

A network is two things: an architecture (the code that builds the layers) and parameters (arrays of numbers). Saving the parameters is the easy part, and each framework has a standard way.

Python
# PyTorch: save only the parameters (a "state dict"), not the Python class
torch.save(model.state_dict(), "digits.pt")

model = DigitNet()                                   # rebuild the architecture first
model.load_state_dict(torch.load("digits.pt", weights_only=True))
model.eval()                                         # inference mode for dropout / batch norm

Keras uses model.save("digits.keras"), as covered in TensorFlow and Keras Essentials. Some safety and robustness rules apply to every framework:

  • Never load a pickle file you do not trust. Python's pickle, which older save formats use, can run arbitrary code when loaded. Prefer weight-only formats such as weights_only=True in PyTorch or the safetensors format.
  • Save the config with the weights: layer sizes, preprocessing constants (mean, standard deviation, vocabulary), class names. A model without its preprocessing gives wrong answers silently.
  • Put model.eval() first (and torch.no_grad() around inference), or dropout and batch norm will still behave as in training.

Exporting to a portable format

Training code depends on a framework, a Python version and often a GPU library. A server may have none of them. Export turns the model into a self-contained graph of operations that a lightweight runtime can execute.

  • TorchScript and torch.export capture a PyTorch model as a graph that runs without your Python class definition, including from C++.
  • ONNX (Open Neural Network Exchange) is a framework-neutral file format. You export from PyTorch, TensorFlow or others, then run it with ONNX Runtime on servers, browsers or phones.
  • TensorFlow SavedModel and TFLite are the TensorFlow family equivalents, the latter aimed at mobile and embedded devices.
Python
import torch

example = torch.zeros(1, 64)                         # a dummy input fixes the graph shape
torch.onnx.export(model, example, "digits.onnx",
                  input_names=["x"], output_names=["logits"],
                  dynamic_axes={"x": {0: "batch"}})  # allow any batch size
Python
import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("digits.onnx")
logits = session.run(None, {"x": np.zeros((5, 64), dtype=np.float32)})[0]

Always test an exported model against the original on a few hundred real inputs. Small numerical differences are normal; large ones mean an operation was captured wrongly, or preprocessing differs.

Quantization

Weights are normally stored as 32-bit floats (4 bytes each). Quantization stores them as small integers instead, most commonly 8-bit integers (int8) with a scale that maps back to real values. The model shrinks by a factor of 4 and integer arithmetic is faster on many processors. Newer large language models go further, with 4-bit weights.

The simplest scheme is symmetric: choose scale = max(|w|) / 127, store q = round(w / scale) clipped to [-127, 127], and recover w โ‰ˆ q * scale. Each weight is off by at most half a step, scale / 2. A single scale for a whole matrix is called per-tensor quantization; using one scale for each output column is per-channel and handles weights of very different sizes better.

The example trains a digit classifier, quantizes its weights, runs inference both with dequantized weights and with genuine integer arithmetic, and then shows why per-channel scales matter when one column has an outlier.

Python Int8 quantization of a trained network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy is unchanged at 0.9194 with dequantized int8 weights and 0.9169 with fully integer matrix products, while the weights shrink from 9,472 to 2,368 bytes. (The 0.92 accuracy is modest because this network trains with plain full-batch gradient descent; the point here is the comparison.) The error check confirms that no weight is off by more than half a quantization step. The outlier example shows the weakness of one scale: a huge value stretches the scale, so every ordinary weight is rounded coarsely (mean error 0.0166 per-tensor against 0.0006 per-channel). Per-channel scales fix that, which is why they are the default in most toolchains.

This is post-training quantization: you quantize a finished model, often using a small calibration set to choose activation ranges. When the accuracy loss is too large, quantization-aware training simulates the rounding during training so the network learns to tolerate it. In PyTorch, torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8) is the one-line starting point; ONNX Runtime and TFLite have their own quantization tools.

Pruning

Most trained networks contain many weights close to zero that contribute almost nothing. Magnitude pruning sets the smallest weights to exactly zero. The model keeps its shape, but it is sparse, and sparse weights compress well on disk (and run faster when the hardware or library exploits sparsity). Pruning is usually followed by a short round of fine-tuning to recover accuracy; here we measure without it, to see how much redundancy there is.

Python Magnitude pruning and file size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy holds up to roughly 50% sparsity (0.914 dense, 0.919 at 50%), slips at 70% to 80%, and collapses beyond 90% (0.75 at 90%, 0.28 at 98%) once the remaining weights cannot carry the task; fine-tuning after pruning would push the usable range higher. Notice also that zeros only save space when the format can exploit them: the plain array file stays at 19,450 bytes at 90% sparsity, and only the compressed file shrinks (18,028 down to 4,209 bytes). Real speedups need sparse kernels or structured pruning, which removes whole neurons or channels and gives a genuinely smaller dense model. A third technique, knowledge distillation, trains a small student network to imitate the output probabilities of a large teacher.

Batch versus online inference

How you run the model depends on when answers are needed:

  • Batch inference scores a large set of inputs on a schedule (tonight's recommendations, a million documents). Optimize for throughput: big batches keep the hardware busy, and nobody waits for a single answer.
  • Online inference answers one request at a time over HTTP or gRPC, often within tens of milliseconds. Optimize for latency. Because GPUs are inefficient with batch size one, servers often use dynamic batching: wait a few milliseconds, group the requests that arrived, run them together.
  • Edge inference runs on the device itself (phone, camera, browser). It avoids network delay and keeps data local, but demands small, quantized models.

A minimal online service wraps the model in a web framework, loads it once at startup (not per request), validates the input shape and returns the prediction. Production systems add a model server (TorchServe, Triton, TensorFlow Serving or a managed cloud endpoint), authentication, rate limits, and health checks.

flowchart LR
    A["Train"] --> B["Export<br/>(ONNX etc.)"]
    B --> C["Serve<br/>(batch or online)"]
    C --> D["Monitor<br/>(drift, latency, errors)"]
    D -->|"degrades"| E["Retrain on new data"]
    E --> B

The deployment loop: a model is never finished, it is monitored and replaced.

Monitoring and drift

Models fail silently. The code keeps running and returns numbers, but the world has moved. Two kinds of change matter:

  • Data drift: the distribution of the inputs shifts (a new camera, a new user population, a changed sensor).
  • Concept drift: the relationship between inputs and the correct answer changes (customer behavior after a price change).

Monitor operational signals (latency, error rate, memory) and model signals: the distribution of inputs, the distribution of predicted classes and confidences, and, when true labels eventually arrive, accuracy. A popular summary of distribution change is the Population Stability Index (PSI): bin a feature, compare the fraction of data per bin between a reference sample and live data, and sum (live - ref) * ln(live / ref). A common rule of thumb reads PSI below 0.1 as stable, 0.1 to 0.25 as a moderate shift, and above 0.25 as a major shift.

Python Detecting drift with PSI

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With no change, PSI is 0.041, comfortably "stable". Every modified scenario scores far above the 0.25 threshold (2 to 4), and PSI needed no labels to say so: you often learn about drift before you learn about errors. But read the other columns too. Dimming the images by 20% shifts the brightness distribution enormously (PSI 4.3) and costs almost no accuracy (0.917 against 0.919), while sensor noise has a smaller PSI and does real damage (accuracy falls to 0.821, and confidence drops). A drift alarm says "look here", not "the model is broken"; pair it with prediction-level signals and, as soon as they arrive, true labels. When the alarm fires, check the pipeline first (a changed unit or a broken feed is the most common cause), then collect fresh labeled data, retrain, and release the new model gradually: first in shadow mode (predictions logged but not used), then as a canary for a small share of traffic, then fully, with the old version ready for rollback.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should you avoid loading model files saved with Python pickle from untrusted sources?

    Unpickling can run code embedded in the file. Use weight-only formats such as safetensors or weights_only=True, and only load files you trust.
  2. 2. What does symmetric int8 quantization store for a weight matrix?

    Each weight becomes round(w / scale) in an 8-bit integer, and multiplying by the scale recovers an approximation of the original value.
  3. 3. Why is per-channel quantization often better than a single per-tensor scale?

    A single scale is set by the largest value in the whole tensor, so ordinary weights lose precision. Separate scales keep each channel's precision.
  4. 4. A 90% pruned model saved as a plain array file is the same size as the dense one. Why?

    Pruning sets weights to zero, but a dense array still stores every zero. Compression, sparse formats or removing whole neurons are needed for real savings.
  5. 5. Live inputs have a different distribution from the training data, but no labels are available yet. What can you still measure?

    Drift in inputs, confidence and predicted classes can be tracked without labels, and it is an early warning that accuracy may be falling.

Summary

  • Save parameters together with the configuration and preprocessing they need; never unpickle untrusted files, and switch to eval mode for inference.
  • Export to a portable format (ONNX, TorchScript, SavedModel, TFLite) so a runtime can serve the model without the training stack, and test the export against the original.
  • Int8 quantization stores integer codes plus a scale, cuts memory by 4x with a small accuracy cost; per-channel scales handle outliers.
  • Pruning zeros out small weights; savings need compression, sparse formats or structured pruning, and usually fine-tuning.
  • Batch inference favors throughput, online inference favors latency, and edge inference needs small models.
  • Monitor input drift (PSI), prediction distributions and, when available, accuracy; roll out new models through shadow and canary stages.

Next: Model Compression.