Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 38 ยท Integration and Differential Equations

Stiff Systems and Higher-Order ODEs

Learn what stiffness means and when to use Radau, BDF or LSODA instead of RK45. Convert higher-order ODEs, model Lotka-Volterra and Lorenz, and use solve_bvp.

Solving ODEs with solve_ivp showed the default workflow: write f(t, y), call solve_ivp, check success. That works beautifully until you meet a problem that makes the default solver crawl, or an equation that does not look like dy/dt = ... at all. This lesson covers both situations.

First comes stiffness: a numerical trap where a fast-decaying part of the solution forces tiny steps long after that part has vanished. Then we turn higher-order equations (like Newton's F = m a) into the first-order systems the solver expects, simulate two famous systems (predator-prey and the Lorenz equations), and finish with boundary value problems, where the conditions are given at both ends of an interval instead of at the start.

What stiffness means

Imagine a quantity that is pulled toward a slowly moving target by a very strong spring. The strong spring pulls it onto the target within a few milliseconds; after that the solution simply follows the slow target. An explicit method such as RK45 computes the next value from the current one only. For a strong spring that is numerically unstable unless the step stays tiny (smaller than the spring's fast time scale), even though the solution is smooth and boring. The solver is limited by stability, not by accuracy.

A problem is called stiff when widely different time scales coexist and you only care about the slow one. Chemical reaction networks, electrical circuits and heat flow are classic sources.

Implicit methods (Radau, BDF) solve an equation at each step, which costs more per step but stays stable with large steps. LSODA is an adaptive wrapper that starts non-stiff and switches to a stiff method when needed.

flowchart TD
    A["Slow solve with RK45?<br/>Very many steps, or failure"] --> B{"Is the problem stiff?"}
    B -->|"no"| C["Loosen tolerances or use DOP853"]
    B -->|"yes, small system"| D["Radau (accurate) or BDF"]
    B -->|"yes, not sure"| E["LSODA switches automatically"]
    D --> F["Optionally supply jac"]
    E --> F

A practical decision path when an ODE solve is slow.

Seeing it happen

The equation y' = -1000 (y - cos t) has a strong pull toward cos t. Count how much work each method needs for the same tolerance:

Python Explicit versus implicit solvers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four methods agree on the answer, but RK45 needs roughly ten times as many function calls as the implicit methods. Its step size is capped by stability, not by the smoothness of the curve. On a real stiff problem with an extreme spread of time scales the gap becomes a factor of thousands, which is the difference between a second and an hour.

A real stiff problem: Robertson's chemical kinetics

Robertson's system is the standard stiff benchmark. Three species react with rate constants spanning nine orders of magnitude. The total amount is conserved, so y1 + y2 + y3 must stay equal to 1, which makes a useful correctness check. We solve it with the implicit methods only; an explicit method would need an impractical number of steps.

Python Robertson's reactions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Species 2 is tiny (around 1e-5), so it is scaled up in the plot to share an axis. The time axis is logarithmic because the interesting events happen on every scale from microseconds to minutes.

Jacobians

Implicit methods need the Jacobian df/dy, the matrix of how each derivative changes with each variable. By default solve_ivp estimates it by finite differences, calling your function several times. If you can write it down, pass it with jac=, a function jac(t, y) returning a matrix (or a constant matrix, or a sparse one for big systems). That saves function calls, makes Newton iterations more reliable, and matters most when the system has hundreds of variables.

Python Supplying an analytic Jacobian

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tip: Do not guess a method up front. Run RK45 on a short time span; if the number of steps explodes or the solver is oddly slow, switch to LSODA or Radau and compare the answers. Agreement between two different methods is a good sanity check.

Higher-order equations become systems

solve_ivp only understands first-order systems. To solve x'' = ..., introduce the velocity as a new variable: let v = x'. Then x' = v and v' = ... is a first-order system of two equations. An equation of order n becomes a system of n equations, with state [x, x', x'', ...] up to order n - 1.

flowchart LR
    A["x'' + 2 z w x' + wยฒ x = 0<br/>(second order)"] --> B["new variable v = x'"]
    B --> C["x' = v"]
    B --> D["v' = -2 z w v - wยฒ x"]
    C --> E["state = [x, v]"]
    D --> E

Rewriting a damped oscillator as two first-order equations.

A damped spring obeys x'' + 2 z w x' + w^2 x = 0, where w is the natural frequency and z the damping ratio. For z < 1 the exact solution is a decaying cosine-like wave, which lets us verify the numerical answer.

Python A damped spring, solved and checked

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lotka-Volterra: predators and prey

The predator-prey model is a two-variable system. Prey x grow on their own and are eaten at a rate proportional to x*y; predators y die off and grow by eating. Its trajectories are closed loops, and there is a quantity H that stays constant along them, which is perfect for judging the solver.

Python Predators and prey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Populations rise and fall out of phase with each other: predators peak after prey, then crash when food runs out. With a loose tolerance, the conserved quantity would visibly drift and the loops would spiral in or out, so this system is a good test of how much accuracy you really get.

The Lorenz system and sensitivity

The Lorenz equations (three variables, a simplified model of convection) are the textbook example of chaos: tiny differences in the starting point grow exponentially. Run the system twice with initial conditions that differ by one part in a billion and watch the distance between the two trajectories.

Python Chaos: the butterfly effect in numbers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gap grows by a roughly steady factor per unit of time (close to a straight line on the log plot): a billionth of a unit at the start becomes visible after about 30 time units. Run it longer and the gap would saturate at the size of the attractor itself, after which the two runs are unrelated. The lesson for numerical work: a chaotic system cannot be predicted far ahead, no matter how tight your tolerance. Even the floating-point rounding in your computer is a perturbation of this size.

Boundary value problems with solve_bvp

In an initial value problem every condition is given at one time. In a boundary value problem (BVP) conditions are given at both ends of an interval, for example the temperature at both ends of a rod. You cannot just march forward, because you do not know the full starting state. solve_bvp(fun, bc, x, y) solves it by iterating on a whole guess for the solution at once:

  • fun(x, y) returns the derivatives. Here y has shape (n, m) (all mesh points at once), so use vectorized NumPy and np.vstack.
  • bc(ya, yb) returns the residuals of the boundary conditions, which must be zero when satisfied; ya and yb are the states at the left and right ends.
  • x is an initial mesh and y an initial guess with shape (n, len(x)).

The solver adapts the mesh and returns sol.sol, a callable for evaluating the answer anywhere.

Python Two boundary value problems

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Bratu problem shows the main catch of BVPs: there may be zero, one, or several solutions, and the solver finds the one nearest your initial guess. Always check result.status (0 means success), plot the solution, and try different guesses if the problem may have more than one answer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What makes a differential equation stiff?

    Stiffness means a fast-decaying component restricts the step size of explicit methods by stability, even when the solution itself is smooth and slowly changing.
  2. 2. You run solve_ivp with the default RK45 and it takes an enormous number of steps on a smooth-looking problem. What should you try?

    An exploding step count on a smooth solution is the classic sign of stiffness. Implicit solvers (Radau, BDF) or the automatically switching LSODA are designed for it.
  3. 3. How do you solve x'' = -x with solve_ivp?

    Introduce v = x' as an extra variable. The function returns [v, -x], a first-order system of two equations.
  4. 4. In solve_bvp, what does the bc(ya, yb) function return?

    bc receives the state at the left and right boundaries and returns how far each condition is from being satisfied, so the solver can drive those values to zero.
  5. 5. Two Lorenz runs start 1e-9 apart and end up completely different. What is the correct conclusion?

    Sensitive dependence on initial conditions is a property of chaotic systems. No tolerance setting can make a long forecast reliable.

Summary

  • Stiff problems mix fast and slow time scales; explicit solvers like RK45 then take huge numbers of tiny steps. Use Radau, BDF or the self-switching LSODA.
  • Supply jac= (a function or constant matrix) when you can, and compare two methods as a sanity check.
  • Turn an order-n equation into a system of n first-order equations by adding x', x'', ... to the state.
  • Conserved quantities, like the Lotka-Volterra invariant or the total mass in Robertson's system, are excellent accuracy checks.
  • Chaotic systems such as Lorenz amplify tiny differences, so far-future predictions are unreliable whatever the tolerance.
  • solve_bvp(fun, bc, x, y) needs a mesh and an initial guess, uses vectorized functions, and may find different solutions for different guesses.

Next up: estimate values between data points in One-Dimensional Interpolation.