Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 45 ยท Autograd

Custom Autograd Functions and Hooks

Write a torch.autograd.Function with its own backward, verify it with gradcheck, use tensor hooks, higher-order gradients and torch.func, with live checks.

Almost everything you need is already differentiable in PyTorch. Sometimes it is not: you need a gradient that is cheaper or more stable than the automatic one, you wrap code that autograd cannot see (a C extension, a lookup table, a numerical solver), or you want a deliberate "lie" in the backward pass, as in straight-through estimators. For those cases PyTorch lets you write your own operation by subclassing torch.autograd.Function and defining the forward and the backward yourself. This lesson shows how, how to prove your backward is correct, and how hooks let you watch or change gradients on the fly.

As in the other autograd lessons, PyTorch code is read-only here. The live editors use NumPy stand-ins: a tiny Function-style class with a ctx object, a finite-difference gradient checker (the same idea as gradcheck), second derivatives by differences, and a hand-written hook. They are not PyTorch, but they let you run the ideas.

When to write your own Function

Stability A safer formula for a tricky gradient.
Foreign code Wrap a solver or extension autograd cannot trace.
Custom rule Straight-through estimators and clipping.
Memory Recompute instead of storing big tensors.
1 / 2

A custom Function in PyTorch

A custom function is a class with two static methods. forward computes the result and may stash tensors on the context object ctx; backward receives the gradient of the loss with respect to the output (grad_output) and must return the gradient with respect to each input. Here is a softplus, log(1 + e^x), with a hand-written backward. Its derivative is the sigmoid.

Python
import torch

class Softplus(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.save_for_backward(x)                    # keep what backward needs
        return torch.logaddexp(torch.zeros_like(x), x)

    @staticmethod
    def backward(ctx, grad_output):
        (x,) = ctx.saved_tensors
        return grad_output * torch.sigmoid(x)       # chain rule: upstream * local

x = torch.randn(5, requires_grad=True)
y = Softplus.apply(x)          # always call .apply, never forward() directly
y.sum().backward()
print(x.grad)

The rules that matter:

  • Call Softplus.apply(...), not forward. apply is what records the node in the graph.
  • backward returns one value per input to forward (return None for inputs that are not tensors or do not need gradients). grad_output has the shape of the output; what you return has the shape of the corresponding input.
  • Use ctx.save_for_backward for tensors you need later (inputs and outputs). It lets PyTorch check that nobody modified them in place. Plain numbers or flags can simply be stored as attributes, ctx.alpha = 0.5.
  • ctx.needs_input_grad tells backward which inputs actually need a gradient, so you can skip wasted work.
  • Inside forward, you are outside the graph. Operations there are not tracked. The gradient is only what your backward says, which is the whole point.
  • Newer code often separates the context handling into a setup_context(ctx, inputs, output) static method and keeps forward pure; this is needed for the function transforms in torch.func. The behavior is the same.

A famous use of a "wrong on purpose" backward is the straight-through estimator: the forward pass rounds a number (whose true gradient is zero almost everywhere), while the backward pass pretends rounding was the identity so that learning can continue.

Python
class RoundSTE(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return torch.round(x)

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output          # pretend d round(x)/dx = 1

Proving your backward is right with gradcheck

A hand-written backward can be silently wrong: the code runs and training "sort of" works. PyTorch ships a tool that compares your analytic gradient with finite differences:

Python
from torch.autograd import gradcheck

x = torch.randn(4, dtype=torch.double, requires_grad=True)   # float64 is essential
print(gradcheck(Softplus.apply, (x,), eps=1e-6, atol=1e-4))   # True, or raises an error

Use float64 inputs: in float32 the finite differences have too much rounding noise and gradcheck fails even for correct code. If the check fails, PyTorch prints both Jacobians so you can see which entries differ. Put a gradcheck call in a unit test for every custom function you write (see the lesson on testing).

The editor below implements the same machinery with NumPy. It defines a Function-like base class and a gradcheck, then tests a correct softplus and a broken one whose backward forgot the sigmoid. Notice that the checker contracts the output with a random weight vector: that way a single number depends on every output element, and the gradient to compare is what real backpropagation would pass back.

Python A Function stand-in with a gradient checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two lines show a very real reason to write careful custom functions: the naive formula overflows to inf for large inputs while the stable form stays finite.

Tensor hooks

A hook is a function PyTorch calls for you at a particular moment. The most useful kind for autograd is the tensor hook: t.register_hook(fn) calls fn(grad) when the gradient with respect to t is computed, during backward(). If fn returns a tensor, it replaces the gradient; if it returns None, the gradient is untouched.

Python
w = torch.tensor([3.0, -4.0], requires_grad=True)
x = torch.tensor([2.0, 5.0])

h = w * x                                  # a non-leaf
h.register_hook(lambda g: print("grad at h:", g))      # peek only
w.register_hook(lambda g: g.clamp(-5.0, 5.0))          # clip what lands in w.grad

loss = (h ** 2).sum()
loss.backward()
print(w.grad)

Hooks are great for debugging ("where does the gradient explode?"), for per-tensor gradient clipping, and for logging gradient norms. Two sibling mechanisms work on whole modules: module.register_forward_hook(fn) sees every input and output of a layer (useful for extracting features from a pretrained network, see Transfer Learning), and module.register_full_backward_hook(fn) sees the gradients flowing through a layer. Each registration returns a handle; call handle.remove() when you are done.

Higher-order gradients

By default backward() throws away the graph that describes the backward pass itself, so you cannot differentiate a gradient. If you pass create_graph=True, PyTorch records the backward computation as a new graph, and you can differentiate again. This is how Hessians, gradient penalties (WGAN-GP) and meta-learning are built.

Python
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
(g1,) = torch.autograd.grad(y, x, create_graph=True)   # 3x^2 = 12
(g2,) = torch.autograd.grad(g1, x)                      # 6x = 12
print(g1.item(), g2.item())                             # 12.0 12.0

torch.autograd.grad(outputs, inputs) returns gradients instead of storing them in .grad, which is handy when you want a gradient without touching the parameters' accumulated values. The newer torch.func module goes further with composable function transforms, in the style of JAX:

Python
from torch.func import grad, vmap, hessian

def f(x):
    return (x ** 3).sum()

x = torch.tensor([1.0, 2.0])
print(grad(f)(x))        # tensor([ 3., 12.])
print(hessian(f)(x))     # diag(6x) = [[6, 0], [0, 12]]

# per-example gradients: vmap maps grad over a batch dimension
def loss(w, x, y):
    return ((x @ w - y) ** 2)
per_sample = vmap(grad(loss), in_dims=(None, 0, 0))(w, X, Y)

grad(f) takes a plain function and returns a function that computes the gradient, with no .backward() or .grad in sight. vmap removes explicit loops over a batch, and hessian, jacrev and jacfwd compute Jacobians and Hessians. The live example below does by finite differences what those transforms do exactly: a second derivative, a Hessian and a hook that changes a gradient in a hand-written backward pass.

Python Second derivatives, a Hessian and a hook, by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  • Prefer composing existing differentiable operations. Write a custom Function only when you can state why autograd cannot do it.
  • Always test with gradcheck in float64, ideally also with a few random shapes.
  • In backward, shapes are everything: the returned gradient must have the shape of the input. If you broadcast in forward, sum in backward (as in Build a Tiny Autograd Tensor).
  • A hook that keeps references to tensors can keep a whole graph alive. Store g.detach() or a Python number.
  • Remove hooks you registered temporarily, or they will fire in every later step.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How do you call a custom torch.autograd.Function so that it is recorded in the graph?

    apply() creates the graph node, runs forward and connects backward. Calling forward() directly bypasses autograd.
  2. 2. Why should you run gradcheck with float64 inputs?

    Finite differences subtract two nearly equal numbers, which loses most of the precision in float32. In float64 the comparison is meaningful.
  3. 3. What must a Function's backward return when forward takes two tensor inputs?

    backward maps grad_output to one gradient per forward input. Use None for an input that needs no gradient.
  4. 4. What is create_graph=True for in torch.autograd.grad?

    Without it the backward computation is not recorded, so second derivatives, Hessian-vector products and gradient penalties are impossible.
  5. 5. A tensor hook registered with t.register_hook(fn) returns None. What happens to the gradient?

    Returning a tensor replaces the gradient; returning None leaves it alone.

Summary

  • Subclass torch.autograd.Function, write static forward and backward, and call .apply; backward returns one gradient per input.
  • Save what backward needs with ctx.save_for_backward; code inside forward is not tracked.
  • Verify every custom function with torch.autograd.gradcheck on float64 inputs; finite differences are the referee.
  • Tensor hooks observe or replace gradients during backward; module hooks do the same for layer inputs, outputs and gradients.
  • create_graph=True makes second derivatives possible; torch.func offers grad, vmap, hessian and Jacobian transforms.

You now know how autograd works from the inside. In the next chapter you use it to fit your first model: Linear Regression the PyTorch Way.