Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Advanced Topics

Mixed Precision and Distributed Training

Train faster with mixed precision (float16, loss scaling) and scale across GPUs with tf.distribute strategies. NumPy demos of float16 limits and data-parallel gradient averaging.

Large models and datasets push training time from minutes to days. Two techniques give big speedups without changing the model's design. Mixed precision does most arithmetic in 16-bit floats, which uses half the memory and runs much faster on modern GPUs and TPUs. Distributed training splits the work across several devices. This lesson explains how each works, what can go wrong, and how to switch them on in Keras.

TensorFlow and GPUs are not available in the browser runner. Keras and tf.distribute code is read-only. The runnable blocks use NumPy to show the numerical facts and to simulate data parallelism.

Floating-point formats

Format Bits Approx. range Digits of precision
float32 32 up to 3.4e38 about 7
float16 16 up to 65,504 about 3
bfloat16 16 up to 3.4e38 about 2

float16 has a small range and low precision. bfloat16 (used by TPUs and recent GPUs) keeps float32's range with even less precision. Look at what that means for ordinary numbers.

Python Limits of float16

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two problems appear: overflow (large values become inf) and underflow (tiny values become 0). Gradients are often very small, so float16 training can silently drop them.

How mixed precision solves it

Keras' mixed_float16 policy keeps two copies of the numbers:

  • Computation in float16: matrix multiplications and convolutions, where speed matters.
  • Variables (weights) stored in float32: the "master weights" that accumulate many tiny updates accurately.
  • Loss scaling: multiply the loss by a large factor (for example 1024) before backpropagation so small gradients stay representable in float16, then divide the gradients by the same factor before the update. Keras's LossScaleOptimizer does this and adjusts the factor automatically.
Python Why loss scaling helps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last line: float16 never moved at all, because each tiny update was rounded away. That is exactly why Keras keeps the master copy of weights in float32.

Using it in Keras

Python
import keras

keras.mixed_precision.set_global_policy("mixed_float16")   # or "mixed_bfloat16"

model = keras.Sequential([
    keras.Input(shape=(784,)),
    keras.layers.Dense(4096, activation="relu"),
    keras.layers.Dense(4096, activation="relu"),
    # the final softmax should stay in float32 for numerical stability
    keras.layers.Dense(10, activation="softmax", dtype="float32"),
])
model.compile("adam", "sparse_categorical_crossentropy", metrics=["accuracy"])

Guidelines:

  • The output layer (softmax, sigmoid, losses) should be float32: pass dtype="float32".
  • With mixed_float16, compile wraps the optimizer in a LossScaleOptimizer automatically; custom loops need to use it explicitly.
  • Speedups appear on GPUs with tensor cores (NVIDIA Volta or newer) and TPUs. On a CPU it is usually slower.
  • Make layer sizes multiples of 8 so tensor cores are used efficiently.
  • Prefer mixed_bfloat16 on hardware that supports it: no loss scaling is needed because its range matches float32.

Distributed training: data parallelism

The most common scheme is data parallelism: every device holds a full copy of the model, each gets a different slice of the batch, computes gradients, and the gradients are averaged across devices (an all-reduce) so all copies apply the identical update and stay in sync.

flowchart TB
    B["Global batch of 64"] --> S["split"]
    S --> G1["GPU 0<br/>batch 32<br/>model copy"]
    S --> G2["GPU 1<br/>batch 32<br/>model copy"]
    G1 --> A["average gradients<br/>(all-reduce)"]
    G2 --> A
    A --> U["same update<br/>on every copy"]

The result equals training on the whole batch on one device, only faster.

For a mean-squared-error loss this equivalence can be checked numerically: the average of the per-device gradients on equal-sized slices equals the gradient on the whole batch.

Python Averaging gradients across devices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

tf.distribute strategies

Python
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()        # all GPUs on one machine
print("replicas:", strategy.num_replicas_in_sync)

global_batch = 64 * strategy.num_replicas_in_sync   # scale the batch with the devices

with strategy.scope():
    model = build_model()                           # variables created here are mirrored
    model.compile("adam", "sparse_categorical_crossentropy", metrics=["accuracy"])

model.fit(train_ds.batch(global_batch), epochs=10)  # fit works unchanged
Strategy Use
MirroredStrategy Several GPUs, one machine (synchronous)
MultiWorkerMirroredStrategy Several machines (synchronous), configured with TF_CONFIG
TPUStrategy Cloud TPUs
ParameterServerStrategy Asynchronous parameter servers (rare today)

Remember:

  • Create the model, optimizer and metrics inside strategy.scope().
  • The batch you pass is the global batch; each replica gets global / n. When you add GPUs, raise the batch size, and usually the learning rate (a common starting rule is to scale it proportionally, with a warm-up).
  • Make the input pipeline fast with prefetching, otherwise extra GPUs wait for data.
  • Keras 3 has its own multi-backend keras.distribution API (data and model parallelism for JAX and TensorFlow) for very large models.
Python Linear learning-rate scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Scaling beyond very large batches stops working without extra care (warm-up, LARS-style optimizers), so treat the rule as a starting point to validate.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In mixed precision training, in which dtype are the master weights kept?

    float32 master weights accumulate many small updates that float16 would round away.
  2. 2. What problem does loss scaling address?

    Scaling the loss makes small gradients representable in float16; they are unscaled before the update.
  3. 3. In MirroredStrategy, what is averaged across replicas?

    Each replica computes gradients on its slice; the average is applied identically everywhere.
  4. 4. Where must the model be built when using a tf.distribute strategy?

    Variables created inside the scope become distributed (mirrored) variables.

Summary

  • float16 is fast but has a small range and low precision; bfloat16 keeps the range.
  • Mixed precision computes in 16-bit, keeps float32 master weights and uses loss scaling (automatic with mixed_float16).
  • Keep the output layer in float32.
  • Data parallelism averages gradients across replicas and equals single-device training on the combined batch.
  • Build the model in strategy.scope(), use a global batch, and keep the input pipeline fast.

Next: Autoencoders and GANs