Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 45 ยท Debugging and Practice

Testing and Deploying Models

Write unit tests for models and losses, check determinism and eval mode, then learn the options for serving, quantizing and exporting a PyTorch model.

A model that trains is not yet a model you can ship. Between the notebook and production there are two jobs: proving that the code does what you think (testing), and turning trained weights into something other programs can call quickly and safely (deployment). Machine-learning code is awkward to test, because results are numbers rather than exact answers, but you can still check a surprising amount: shapes, invariants, determinism, the behavior of the loss and whether the model can learn at all.

PyTorch itself cannot run on this site, so its code here is read-only. The live editors use NumPy stand-ins: a tiny network with a hand-written test suite (and a runner that prints PASS and FAIL lines the way pytest does), and a small int8 quantization and serving demo. Where a stand-in differs from the real thing, the text says so. You should know debugging PyTorch code and saving and loading models first.

What to test in a model

Shapes Output shape for any batch size.
Determinism Eval mode gives identical outputs.
Loss Known values and no NaN on extreme inputs.
Can it learn? Overfit one tiny batch.

None of these needs a trained model or a big dataset.

1 / 2

What a good model test looks like

Tests for model code follow the same rules as any other tests: small, fast, deterministic, and each one checking one idea. Use tiny dimensions (a hidden size of 16, not 4,096), random data from a fixed seed, and no downloads. The most valuable checks are cheap:

  • Shape tests. Feed a batch and assert the output shape. Do it for batch size 1 as well, because code that squeezes a dimension often breaks only there.
  • Invariant tests. Without dropout or batch statistics, row i of the output must not depend on the other rows in the batch. If it does, something is mixing examples (a common bug in attention masks and in batch norm used the wrong way).
  • Loss tests. Uniform logits over C classes must give a loss of log(C). A loss on logits like 1000 must stay finite. These two catch most numerical mistakes (see Activation and Loss Functions).
  • Gradient flow tests. After one backward pass, every parameter should have a gradient that is not None and not all zeros (a frozen or detached branch shows up here).
  • Overfit-one-batch test. A model with enough capacity must be able to drive the loss on 8 samples close to zero. If it cannot, there is a bug in the model, the loss or the optimizer wiring, whatever your dataset looks like.

With PyTorch and pytest, such tests look like this (read-only):

Python
import torch
import pytest
from torch import nn

from mypackage.model import Net

@pytest.fixture
def batch():
    g = torch.Generator().manual_seed(0)
    return torch.randn(8, 10, generator=g), torch.randint(0, 3, (8,), generator=g)

def test_output_shape(batch):
    x, _ = batch
    assert Net(10, 3)(x).shape == (8, 3)
    assert Net(10, 3)(x[:1]).shape == (1, 3)         # batch of one

def test_every_parameter_gets_a_gradient(batch):
    x, y = batch
    model = Net(10, 3)
    nn.functional.cross_entropy(model(x), y).backward()
    for name, p in model.named_parameters():
        assert p.grad is not None, name
        assert p.grad.abs().sum() > 0, name

def test_overfits_one_batch(batch):
    x, y = batch
    torch.manual_seed(0)
    model = Net(10, 3)
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    for _ in range(300):
        opt.zero_grad()
        loss = nn.functional.cross_entropy(model(x), y)
        loss.backward()
        opt.step()
    assert loss.item() < 0.05

For comparing tensors use torch.testing.assert_close(actual, expected), which applies sensible tolerances for each dtype and prints a readable report of the mismatch. Never compare floating-point results with == unless you know they are bit-identical.

Determinism and eval mode

A test that sometimes fails is worse than no test, so control the randomness. torch.manual_seed(0) seeds the default generator; for data pipelines pass a torch.Generator explicitly, as in the fixture above. The training loop is a different matter: on a GPU some kernels are non-deterministic for speed, and torch.use_deterministic_algorithms(True) makes PyTorch raise an error when you reach one. Test on CPU where possible, and compare with a tolerance instead of exact equality.

The second source of "randomness" is not random at all: mode. In train() mode dropout drops units and batch norm uses the statistics of the current batch; in eval() mode dropout is the identity and batch norm uses its stored running averages (see Initialization, Normalization and Dropout). Forgetting model.eval() is the classic reason a deployed model gives slightly different answers on every call. A test should therefore assert that two eval-mode calls return identical outputs, and that train-mode calls with dropout do not, which proves the switch really works.

The editor below is a stand-in test suite. TinyNet is a two-layer network in NumPy with a dropout switch that mimics train() and eval(). The suite runs once with a numerically stable cross-entropy and once with the naive one, which computes exp of the raw logits. The naive version overflows on huge logits, and one test should catch it.

Python A model test suite with a runner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first run prints 7/7 passed. The second prints 6/7 passed, and the failing line is test_loss_survives_huge_logits: the naive loss computes exp(1000), which overflows to infinity, and the division gives nan. That is the whole point of a test suite: the bug is named by the failing test instead of showing up as a loss that turns into nan three hours into training. Notice that the other six tests pass for both versions, so each test checks one idea and the failure points at one cause.

The stand-in does not run PyTorch, so it cannot show gradient-flow checks or assert_close. Everything else (the structure, the seeded data, the independence and mode checks, the overfit test) carries over unchanged.

Serving a model for inference

Once the weights are trusted, serving is mostly about doing less work, not more. A few rules cover most of it:

Python
import torch

model = Net(10, 3)
model.load_state_dict(torch.load("model.pt", weights_only=True))
model.eval()                                  # dropout off, batch norm uses running statistics

@torch.inference_mode()                       # no graph, no gradients, less memory
def predict(batch: torch.Tensor) -> torch.Tensor:
    return model(batch).softmax(dim=-1)
  • Load once. Build the model and load the weights when the process starts, not per request.
  • eval() and inference_mode(). They are different things: eval() changes how layers behave, inference_mode() (or no_grad()) stops PyTorch from recording a graph. You want both. See Autograd in Practice.
  • Validate input before the model sees it. Check the shape, the dtype and that no value is NaN or infinite, and answer with a clear error. A model never raises an exception for nonsense input; it just returns nonsense.
  • Batch requests. One forward pass on 32 inputs is far cheaper than 32 passes on one input each, especially on a GPU. Serving frameworks collect requests for a few milliseconds and run them together (dynamic batching).
  • Keep preprocessing identical to training. The same normalization statistics, the same tokenizer, the same image size. Mismatched preprocessing is a more common cause of bad predictions in production than a bad model.
  • Version everything. Store the weights, the architecture code version and the preprocessing settings together, and log which version produced each prediction.

A web service wraps predict in an HTTP framework (FastAPI is a popular choice) or in a dedicated model server such as TorchServe or NVIDIA Triton. On a single CPU box a plain Python process with a few worker processes is often enough; do not reach for a cluster before you have measured.

Making the model smaller: quantization

A trained network stores millions of 32-bit floats. Quantization stores the weights (and sometimes the activations) as 8-bit integers instead, which cuts memory to a quarter and can speed up inference on hardware with fast integer math. The simplest scheme is symmetric: pick one scale so that the largest weight maps to 127, store round(w / scale) as an int8, and recover approximate weights with q * scale. Each weight is off by at most scale / 2.

PyTorch offers several routes: dynamic quantization of nn.Linear layers (weights int8, activations quantized on the fly), static quantization with a calibration pass over sample data, and quantization-aware training, which simulates rounding during training so the model learns to tolerate it. The details of those APIs change between releases, so check the documentation of your version. What does not change is the trade-off, which the next editor measures with a NumPy stand-in on one layer with random weights. It also includes a tiny predict function with the input validation described above.

Python int8 quantization and request validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The int8 weights take 2,048 bytes instead of 8,192, exactly a quarter. The largest weight error equals half the scale, as predicted, and the outputs differ by under one percent in relative terms. The top class agrees on 99.2% of the 500 random inputs. That is typical: most inputs are far from a decision boundary, and a few near one flip. Real quantization work always ends with the same step: measure the accuracy of the quantized model on a real validation set before you ship it.

The predict function at the bottom answers bad requests with a message instead of crashing or, worse, returning a confident answer for garbage. This shape (validate, run, format) is the core of every inference endpoint, whatever the framework.

Exporting a model

Sometimes the consumer is not Python. Exporting writes the model into a format that other runtimes can load:

Python
example = torch.randn(1, 10)

# ONNX: an open format, runs in ONNX Runtime from C++, C#, Java, JavaScript...
torch.onnx.export(model, (example,), "model.onnx",
                  input_names=["features"], output_names=["logits"],
                  dynamic_axes={"features": {0: "batch"}})   # allow any batch size

# TorchScript: a serialized, Python-free form of the model that LibTorch can load
scripted = torch.jit.trace(model, example)
scripted.save("model.ts")
Option Good for Watch out for
Keep PyTorch (state_dict) Python services, fastest to build Needs the code and PyTorch installed
ONNX Other languages, mobile, optimized runtimes Some operations or dynamic control flow may not convert
TorchScript C++ apps using LibTorch trace records one path and ignores if statements on data
torch.compile Faster Python inference Still needs Python and PyTorch

Tracing runs the model once on the example input and records the operations, so any data-dependent if or loop is frozen into the path that example took. The fix is to export with a representative input and then test the exported model against the original: run both on the same batch and compare with a tolerance. That check belongs in your test suite next to the others, for example with onnxruntime and numpy.testing.assert_allclose(original, exported, atol=1e-5).

A deployment checklist

  1. Tests pass: shapes, determinism, loss values, overfit-one-batch.
  2. The model is in eval() mode and runs under inference_mode().
  3. Preprocessing is shared code between training and serving, not copied.
  4. Weights are loaded with weights_only=True or a safe format (saving and loading).
  5. Inputs are validated, and there are timeouts and size limits.
  6. Accuracy of the shipped artifact (quantized or exported) is checked on real validation data.
  7. Latency and memory are measured under realistic batch sizes (Making Training Faster shows how to benchmark).
  8. Every prediction logs the model version, so a bad release can be traced and rolled back.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A deployed model returns slightly different probabilities each time for the same input. What is the most likely cause?

    In train() mode dropout randomly drops units on every forward pass. eval() makes dropout the identity and batch norm use stored statistics, which makes outputs repeatable.
  2. 2. What does the overfit-one-batch test prove when it passes?

    Driving the loss on a handful of samples to nearly zero only needs a correct forward pass, loss, gradients and update. It says nothing about generalization, but a failure here means a bug.
  3. 3. Symmetric int8 quantization uses a scale of 0.0128 for a weight tensor. What is the largest possible error for one weight?

    Each weight is rounded to the nearest multiple of the scale, so the error is at most half of it: 0.0064.
  4. 4. Why should an exported ONNX or TorchScript model be compared with the original on the same inputs?

    A trace records only the path taken by the example input. Comparing outputs with a tolerance catches conversions that silently changed behavior.

Summary

  • Test models with small, seeded, fast tests: output shapes (including batch size 1), row independence, loss values, gradient flow and the overfit-one-batch test.
  • Eval-mode outputs must be identical on repeated calls; train-mode dropout must not be. Forgetting eval() is a classic deployment bug.
  • Compare tensors with torch.testing.assert_close, never with ==.
  • Serve with eval() and inference_mode(), load once, validate every input, batch requests and reuse the training preprocessing.
  • Quantization trades a small, measurable accuracy loss for a smaller and often faster model; always measure on real validation data.
  • Export to ONNX or TorchScript for non-Python consumers, then test the exported model against the original.

Continue with Project: A Neural Network Without PyTorch.