Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Data Pipelines and Saving Models

Deploying Models

Take a trained Keras model to production: export a SavedModel, serve it over REST with TensorFlow Serving, convert to TFLite, and understand quantization with NumPy demos.

A model that lives in a notebook helps nobody. Deployment means packaging the trained model so other programs, phones or web services can call it quickly and reliably. The right choice depends on where it runs: a server answering requests, a mobile or embedded device, or a browser. This lesson explains the main routes and the one idea that makes small devices possible: quantization.

TensorFlow, TensorFlow Serving and TFLite are not available in the browser runner. Their commands and code are read-only; NumPy blocks demonstrate quantization and request handling.

Choosing a target

Where will the model run?

Server TF Serving or a Python API
Phone or device TensorFlow Lite
Browser TensorFlow.js
Managed cloud Vertex AI and similar

All of them start from a saved model.

1 / 2

Step 1: export a SavedModel

Python
model.export("export/my_model/1")      # the trailing 1 is the version number

The directory holds a graph with named inputs and outputs, independent of your Python classes. Use the folder name 1, 2, ... for versions, because TensorFlow Serving reads the highest number and can switch to a new version without downtime.

Inspect the signature from the command line:

TEXT
saved_model_cli show --dir export/my_model/1 --all

Option A: TensorFlow Serving

TensorFlow Serving is a production server (C++, batching, versioning) that loads SavedModels. The usual way to run it is Docker:

TEXT
docker run -p 8501:8501 \
  -v "$PWD/export/my_model:/models/my_model" \
  -e MODEL_NAME=my_model \
  tensorflow/serving

Then call it with JSON over REST (port 8501) or gRPC (8500):

Python
import requests, json

payload = {"instances": [[5.1, 3.5, 1.4, 0.2]]}
r = requests.post("http://localhost:8501/v1/models/my_model:predict",
                  data=json.dumps(payload))
print(r.json()["predictions"])

The request body contains a batch under "instances". The reply has one prediction per instance.

Python Request and response format

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Smaller projects often wrap the model in a Python web framework (FastAPI, Flask): load once at startup, then call model.predict per request. Load the model once, not per request, and batch if you can.

Option B: TensorFlow Lite

TFLite is a compact runtime for Android, iOS, microcontrollers and Raspberry Pi. You convert the model into a .tflite flatbuffer file:

Python
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]   # enables quantization
tflite_model = converter.convert()

with open("model.tflite", "wb") as f:
    f.write(tflite_model)

# test on the desktop
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
inp = interpreter.get_input_details()[0]
out = interpreter.get_output_details()[0]
interpreter.set_tensor(inp["index"], x[:1].astype("float32"))
interpreter.invoke()
print(interpreter.get_tensor(out["index"]))

Apps ship the file and run it with the TFLite interpreter on the device. Export options evolve between releases, so check the documentation of your installed version.

Quantization: smaller and faster

A float32 weight takes 4 bytes. Quantization stores weights as 8-bit integers (1 byte) plus a scale, making the model about four times smaller and often faster on phone CPUs, usually with a small accuracy loss. The mapping is linear: q = round(w / scale), and w โ‰ˆ q * scale.

Python 8-bit quantization of weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error per weight is at most half a quantization step, and these errors average out across a dot product, which is why accuracy often barely changes. Variants:

Mode What changes Typical use
Dynamic range Weights int8, activations computed in float Easiest, no data needed
Full integer Weights and activations int8 (needs a small representative dataset to calibrate) Microcontrollers, accelerators
Float16 Weights stored as half precision GPU delegates, half the size
Quantization-aware training Simulate rounding during training Best accuracy at 8 bits or lower
Python Why calibration data matters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One outlier stretches the range so every ordinary value gets coarser steps. Representative data lets the converter pick better ranges.

Option C: the browser

tensorflowjs_converter turns a SavedModel into JSON plus binary weight shards that TensorFlow.js loads in a web page, so inference runs client-side with no server and keeps user data on the device. Models must be small, because users download them.

Production checklist

  • Preprocess inside the model (Rescaling, TextVectorization, Normalization) so training and serving cannot disagree about scaling.
  • Version models and keep the previous one for rollback.
  • Test the exported artifact, not only the Keras object: compare its outputs with the original on a sample set.
  • Measure latency and memory on the real target hardware.
  • Monitor input distribution and prediction quality after release; data drifts.
  • Batch requests on servers; cap input sizes.
Python Compare exported and original outputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the SavedModel folder for TF Serving end in a number like 1?

    Serving watches the base directory and loads the highest numbered version.
  2. 2. Approximately how much smaller is an int8-quantized model compared with float32?

    4 bytes per weight become 1 byte, so about four times smaller (plus small overhead).
  3. 3. What is the purpose of a representative dataset in full-integer quantization?

    Activation ranges are not stored in the weights, so sample data is run through the model to find them.
  4. 4. Where should input scaling (such as Rescaling or Normalization) live for safest deployment?

    Keeping preprocessing in the model prevents training/serving skew.

Summary

  • Export with model.export(...) to get a SavedModel; use numbered version folders.
  • TensorFlow Serving offers REST/gRPC, batching and version switching; TFLite targets mobile and embedded; TensorFlow.js runs in browsers.
  • Quantization stores weights as int8 for about 4x smaller models with little accuracy loss; calibrate with representative data.
  • Put preprocessing inside the model and verify the exported model against the original.
  • Measure latency on the target device and monitor after release.

Next: Custom Layers and Models