Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Numerical Differentiation and Integration

Numerical Differentiation: Finite Differences

Approximate derivatives with forward, backward and central differences. See truncation versus round-off error, the best step size and complex-step tricks.

Sometimes you cannot differentiate by hand: the function is a black box, a simulation, or a table of measurements. Numerical differentiation estimates f'(x) using only function values. The idea comes straight from the definition of the derivative, replacing the limit by a small but finite step h.

The surprise of this topic is that making h smaller helps only up to a point. Below it, floating-point round-off takes over and the answer gets worse. Understanding that trade-off is the heart of this lesson.

Forward, backward and central differences

Taylor series give the three basic formulas.

  • Forward: f'(x) ~ (f(x + h) - f(x)) / h, error O(h).
  • Backward: f'(x) ~ (f(x) - f(x - h)) / h, error O(h).
  • Central: f'(x) ~ (f(x + h) - f(x - h)) / (2h), error O(h^2).

The central difference is better because the h^2 terms of the two Taylor expansions cancel in the symmetric combination. It costs two evaluations, just like the forward formula costs two (f(x) is usually already known, though, so forward is cheaper when f(x) is available). For the second derivative, the standard stencil is

f''(x) ~ (f(x + h) - 2 f(x) + f(x - h)) / h^2, error O(h^2).

Python Compare the three formulas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Halving h halves the forward and backward errors (first order) but cuts the central error by about four (second order).

Truncation error versus round-off error

Two errors fight each other:

  • Truncation error comes from cutting the Taylor series. It shrinks as h shrinks: O(h^2) for the central formula.
  • Round-off error comes from subtracting two nearly equal numbers. The values f(x + h) and f(x - h) carry relative errors near machine epsilon (about 1e-16); dividing their tiny difference by 2h amplifies that noise like eps / h.

Total error is roughly C1 h^2 + C2 eps / h. It is minimized near h ~ eps^(1/3), about 1e-5 for the central formula, and for the forward formula near eps^(1/2), about 1e-8.

flowchart LR
  A["Large h:<br/>truncation error dominates"] --> B["Sweet spot:<br/>h around 1e-5 (central)"]
  B --> C["Tiny h:<br/>round-off error dominates"]
Python The V-shaped error curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the forward error bottom out around h = 1e-8 and the central error around 1e-5 to 1e-6, then both climb. At h = 1e-15 there is almost no correct digit left. In practice, choose h = sqrt(eps) * max(|x|, 1) for forward differences and eps^(1/3) * max(|x|, 1) for central ones.

Never pick h as small as possible. A step of 1e-15 is typically far worse than 1e-5.

Higher-order stencils

You can get better accuracy by using more points. The five-point central formula

f'(x) ~ (-f(x + 2h) + 8 f(x + h) - 8 f(x - h) + f(x - 2h)) / (12 h)

has error O(h^4). In general, you pick weights so that the Taylor terms up to a chosen order cancel. NumPy's np.gradient applies second-order differences on a grid, and scipy.differentiate.derivative selects steps and orders automatically.

Python Five-point stencil and np.gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the weaker accuracy at the boundaries: one-sided formulas are needed there, and np.gradient uses a first-order one by default (pass edge_order=2 for second order).

Differentiating noisy data

Differentiation amplifies noise. If measurements have error delta, a central difference with spacing h has noise of order delta / h. So with real data:

  • do not use the smallest spacing; smooth first (a least-squares fit, or a smoothing spline, see least squares fitting), then differentiate the smooth curve;
  • or fit a local polynomial in a moving window (Savitzky-Golay filters, scipy.signal.savgol_filter, do exactly this).

A trick: the complex step

For analytic functions evaluated with complex arithmetic, f'(x) ~ Im(f(x + i h)) / h has no subtraction at all, so h = 1e-30 works and the result is accurate to machine precision. It needs code that accepts complex numbers.

Python Complex step derivative

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where an exact derivative is available, prefer automatic differentiation or a symbolic derivative; numerical differentiation is the fallback for black-box functions and measured data. The same finite-difference ideas reappear in Euler's method for ODEs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the order of accuracy of the central difference (f(x+h) - f(x-h)) / (2h)?

    The symmetric combination cancels the h^2 Taylor terms, leaving an error proportional to h^2.
  2. 2. Why does the error grow when h becomes extremely small?

    The rounding noise in f is divided by a tiny h, so the error grows like eps / h.
  3. 3. Roughly which step size minimizes the error of the central difference in double precision?

    Balancing h^2 against eps / h gives h of order eps^(1/3), about 1e-5.
  4. 4. What should you do before differentiating noisy measurements?

    Differentiation amplifies noise, so smoothing first gives a far more stable derivative.

Summary

  • Forward and backward differences are O(h); the central difference and the second-derivative stencil are O(h^2).
  • Total error = truncation (falls with h) + round-off (grows as h falls), giving a best h, near 1e-8 (forward) and 1e-5 (central).
  • Wider stencils raise the order; np.gradient handles tabulated data.
  • Noise is amplified, so smooth first. The complex step avoids subtraction for analytic functions.

Next: Richardson extrapolation and Romberg integration