Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 40 ยท Tensors and Automatic Differentiation

Automatic Differentiation

Understand automatic differentiation: how tf.GradientTape records operations and computes gradients, plus a runnable reverse-mode autodiff built from scratch.

Training needs the gradient of the loss with respect to every weight. Writing those derivatives by hand, as we did for the XOR network, is error-prone and impossible to maintain once models have hundreds of layers. Automatic differentiation (autodiff) does it for you: the library records the operations you run and then replays them backwards, applying the chain rule at each step. In TensorFlow the recorder is called tf.GradientTape.

This lesson shows how to use the tape, then builds a small reverse-mode autodiff engine in NumPy-free Python so you can see exactly what happens inside.

Gradients in one paragraph

For a function y = f(x), the derivative dy/dx says how much y changes per tiny change in x. For y = x**2, dy/dx = 2x, so at x = 3 the slope is 6. With many inputs, the gradient is the vector of partial derivatives, one per input. For a loss that depends on thousands of weights, the gradient tells you which direction to move every weight to reduce the loss fastest.

tf.GradientTape

Python
import tensorflow as tf

x = tf.Variable(3.0)

with tf.GradientTape() as tape:
    y = x ** 2            # operations are recorded

dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())      # 6.0

The tape records every operation involving watched tensors while inside the with block. Then tape.gradient(target, sources) runs backwards through the record. Details worth knowing:

  • Variables are watched automatically (if trainable). For a plain tensor, call tape.watch(t).
  • A tape is used once by default. After one call to tape.gradient its resources are freed. For several gradients from one recording, use tf.GradientTape(persistent=True) and delete the tape when done.
  • Sources can be a list; the result is a list in the same order.
  • If the source is not connected to the target, the gradient is None, a hint that something broke the chain (for example converting to NumPy in the middle).
Python
w = tf.Variable([[1.0], [2.0]])
b = tf.Variable([0.5])
x = tf.constant([[3.0, 4.0]])

with tf.GradientTape() as tape:
    y_hat = x @ w + b                    # shape (1, 1)
    loss = tf.reduce_sum(y_hat ** 2)

grad_w, grad_b = tape.gradient(loss, [w, b])

Here y_hat = 3*1 + 4*2 + 0.5 = 11.5, and loss = y_hat**2. The gradient with respect to w is 2 * y_hat * x^T = 2*11.5*[3, 4] = [69, 92] and with respect to b is 2 * 11.5 = 23. So grad_w is [[69.], [92.]] and grad_b is [23.].

This lesson's code blocks that use tf. are read-only because TensorFlow does not run in the browser. The runnable example below builds the same machinery from scratch.

Higher-order and per-example gradients

Nest tapes to get second derivatives, which some advanced methods use:

Python
x = tf.Variable(2.0)
with tf.GradientTape() as outer:
    with tf.GradientTape() as inner:
        y = x ** 3
    dy = inner.gradient(y, x)       # 3x^2 = 12
d2y = outer.gradient(dy, x)         # 6x   = 12

For per-example gradients use tape.jacobian. Both are rarely needed in everyday Keras work, where fit does all this behind the scenes.

How the chain rule works

Autodiff builds a computation graph. Take y = (a * b) + c. Each operation node knows its own local derivative. To get dy/da you multiply local derivatives along the path from y back to a.

flowchart LR
    a["a"] --> m["multiply"]
    b["b"] --> m
    m -->|"p = a*b"| s["add"]
    c["c"] --> s
    s --> y["y = p + c"]

Forward: values flow left to right. Backward: gradients flow right to left, multiplied by local derivatives.

Backward: dy/dy = 1. The add node passes that 1 to both p and c. The multiply node gives a the gradient 1 * b, and b the gradient 1 * a. When a value is used more than once, gradients from each use are added.

This is reverse-mode autodiff, and it is what makes training feasible: one backward pass yields the gradient for all weights, no matter how many there are, at a cost similar to one forward pass.

Autodiff from scratch

Here is a tiny engine. A Value stores a number, its gradient, and a function that pushes gradient to its parents. Study __mul__ and backward: the chain rule is only a few lines.

Python Stand-in: reverse-mode autodiff in plain Python

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers match what tf.GradientTape returns for the same computation. TensorFlow's version works on whole tensors and thousands of operation types, but the principle is identical.

Checking gradients numerically

How do you trust a gradient? Compare it with a finite difference: nudge the input by a tiny h and measure the change in output: (f(x+h) - f(x-h)) / (2h). This is slow (one function evaluation per parameter) but excellent for tests. Frameworks and researchers use it to verify custom gradients.

Python Stand-in: gradient checking with finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A tiny difference (around 1e-9 to 1e-10) means the formula is right. A large one means a bug.

Stopping and controlling gradients

  • tf.stop_gradient(t) treats t as a constant in the backward pass.
  • Wrap code in with tf.GradientTape(watch_accessed_variables=False) to choose exactly what is tracked.
  • Non-differentiable operations (rounding, argmax, integer casts) have no useful gradient, which is why models are trained on a smooth loss (cross-entropy) even if you report accuracy.
  • Exploding or vanishing gradients (very large or tiny values) are a training problem, not an autodiff problem; see Chapter 4.

Where this appears in Keras

When you call model.fit, Keras runs a loop that does exactly this for each batch: open a GradientTape, compute predictions and loss, ask for gradients with respect to model.trainable_weights, and hand them to the optimizer. Gradient Descent from Scratch assembles that loop.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does tf.GradientTape do?

    Operations inside the tape are recorded; tape.gradient then applies the chain rule in reverse.
  2. 2. Why is reverse-mode autodiff used for neural networks?

    With a scalar loss and many parameters, reverse mode is efficient: a single backward pass gets every gradient.
  3. 3. tape.gradient returns None for a variable. What is the most likely cause?

    Something broke the chain, for example calling .numpy() or using a non-differentiable op, or the computation happened outside the tape.
  4. 4. In the from-scratch engine, why does backward use self.grad += ... instead of =?

    By the multivariable chain rule, contributions from every path to a value add up.

Summary

  • Autodiff computes exact gradients by recording operations and applying the chain rule backwards.
  • tf.GradientTape records; tape.gradient(loss, variables) returns gradients; variables are watched automatically.
  • Gradients of a shared value add; non-differentiable operations break the chain.
  • A 40-line Value class reproduces the result, and finite differences check it.
  • Keras fit uses this machinery every batch.

Next: Gradient Descent from Scratch