Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 73 ยท Neural Networks From Scratch

Gradient Checking

Verify a hand-written backward pass with finite differences: centered differences, relative error, per-parameter checks, tolerances, and finding a bug.

A hand-written backward pass is easy to get subtly wrong. A missing transpose, a forgotten 1/N, or a gradient that is correct for one layer and off by a factor of two for another will not crash anything. The network will simply train badly, and nothing in the loss curve tells you why. Gradient checking is the cure: compare your analytic gradient with a slow, brute-force estimate that cannot be wrong in the same way.

The brute-force estimate is based on the definition of the derivative. It costs two extra forward passes per parameter, so you use it only while developing, on a tiny network and a handful of examples. Frameworks with automatic differentiation make hand-written backward passes unnecessary, but while you write one yourself (as in Backpropagation in NumPy), this is the most valuable debugging tool you have.

The idea in three steps

Nudge Change one parameter by a tiny amount, up and down.
Measure See how much the loss moved.
Compare Match it against your backprop gradient.

Cheap to write, slow to run, nearly impossible to fool.

1 / 2

Finite differences

By definition, the derivative of f at x is the limit of (f(x + h) - f(x)) / h as h shrinks to zero. If you pick a small but finite h, you get an approximation. The forward difference uses that formula directly. The centered difference looks on both sides:

TEXT
f'(x) ~ ( f(x + h) - f(x - h) ) / (2h)

The centered version is far more accurate: its error shrinks with h squared instead of h. Compare the two on f(x) = sin(x) at x = 1, where the true derivative is cos(1).

Python Forward versus centered differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two effects fight each other. Large h makes the formula a poor approximation of the slope (truncation error). Tiny h makes f(x + h) and f(x - h) almost equal, and subtracting nearly equal floating-point numbers destroys precision (round-off error). The centered curve bottoms out around h = 1e-5 to 1e-6, which is why gradient checkers use h near 1e-5 and always use 64-bit floats.

Relative error

How do you decide that two gradients "match"? An absolute difference of 1e-4 is huge if the gradient is 1e-6 and nothing if it is 1000. Use the relative error:

TEXT
rel_error = |g_analytic - g_numeric| / max( |g_analytic| + |g_numeric|, tiny )

Often you compute this for the whole gradient array at once, with norms: norm(a - n) / (norm(a) + norm(n)). Typical interpretation with centered differences in float64:

Relative error Verdict
below 1e-7 Correct
1e-7 to 1e-5 Probably fine, but look closer if the network is deep
1e-5 to 1e-3 Suspicious; check for a kink (see below) or a small bug
above 1e-3 Almost certainly a bug

A reusable checker

Here is a small two-layer network with a tanh hidden layer and a softmax-cross-entropy loss. Parameters are kept in one dictionary, so a single function can walk through them. The checker perturbs every number in every array, one at a time, and compares with the backprop gradient.

Python Checking every parameter of a small network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every parameter array reports an error near 1e-10 or 1e-9, far below the 1e-7 line. The checker works on the parameters, not the activations, because the parameters are what training updates.

Finding a bug

Now break the backward pass on purpose, the way a real bug would. In the next example the hidden-layer gradient forgets that tanh has a derivative: the line d_z1 = d_h * (1 - h ** 2) is replaced by d_z1 = d_h. Only the first layer's gradients are affected, since they are computed from d_z1. Watch how the report points directly at the wrong layer.

Python A bug that the checker pinpoints

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output layer (W2, b2) passes, while W1 and b1 fail with relative errors around 1e-1. Because gradients flow backward, a mistake in one place corrupts everything earlier in the network but nothing after it. The first failing layer counted from the loss is therefore where the bug lives. Check layers one by one from the output backward and stop at the first failure.

Practical advice and pitfalls

  • Use float64. In 32-bit floats the round-off floor is around 1e-4, and every check looks suspicious.
  • Use a tiny network and a few examples. A hidden layer of five units is plenty to expose a bug.
  • Check with the real loss. Test the whole chain, including regularization terms, exactly as you train.
  • Beware of kinks. ReLU and max pooling are not differentiable at their corners. If a pre-activation is within eps of zero, the two sides of the finite difference disagree and you see a false alarm. Re-run with another seed or a slightly different eps before suspecting the code.
  • Turn off randomness. Dropout masks and other random choices must be fixed (use the same seed or disable them) or the loss changes between the two evaluations.
  • Check at random points, not just at the initial weights. Some bugs hide when biases are zero.
  • Do not leave it on. It is a development tool. Once the layers pass, remove it from your training loop.

Gradient checking tests that the backward pass matches the forward pass. It cannot tell you whether the forward pass computes what you meant. A wrong loss formula can pass a gradient check perfectly well.

You now have a verified backward pass. In the next lesson, Your First Trained Network, you will use it to train a network on a problem no straight line can solve.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the centered difference (f(x+h) - f(x-h)) / 2h preferred over the forward difference?

    The symmetric formula cancels the leading error term. With h around 1e-5 it is typically accurate to many more digits than the forward difference.
  2. 2. A gradient check on a 5-layer network gives relative errors of 1e-9 for layers 5 and 4 and 0.4 for layer 3. Where is the bug most likely to be?

    Layers closer to the loss are correct. A wrong backward step corrupts everything earlier in the chain, so the first failing layer counted from the output is the culprit.
  3. 3. Why should gradient checks be done in 64-bit floats?

    The numerator subtracts two nearly equal losses, and 32-bit precision leaves too few reliable digits for a trustworthy comparison.
  4. 4. A relative error of 3e-3 appears only on a few ReLU weights. What is a sensible first step?

    At the corner of a ReLU the finite difference averages two different slopes. If the error disappears with a different seed or eps, the code is fine.

Summary

  • Gradient checking compares the analytic gradient with a finite-difference estimate: nudge a parameter up and down and measure the loss change.
  • Use the centered difference with h near 1e-5 and float64 numbers.
  • Judge agreement with relative error; below 1e-7 is correct, above 1e-3 is a bug.
  • Check every parameter array and work backward from the output: the first failing layer is where the bug is.
  • Watch for kinks, dropout randomness and 32-bit precision, and never leave the check in a real training loop.

Next lesson: Your First Trained Network.