Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Data and Science

Differential Equations and Optimization

Solve ODEs with DifferentialEquations.jl, formulate linear programs with JuMP, and minimize functions with Optim.jl, with Python stand-ins you can run.

Two kinds of problems show up again and again in science and engineering. The first asks how something changes over time: a cooling cup of coffee, a swinging pendulum, rabbits and foxes. These are differential equations. The second asks for the best choice among many: the cheapest mix of ingredients, the parameters that fit the data best. These are optimization problems. Julia's ecosystem is especially strong in both, and a good part of the reason people adopt the language is DifferentialEquations.jl, JuMP and Optim.jl.

In this lesson you will set up and solve an ordinary differential equation (ODE), formulate a small linear program, and minimize a function without constraints. These are packages, and Julia is not installed on this site, so the Julia code is read-only. The runnable examples are Python stand-ins: a hand-written Euler method to show what a solver does inside, and SciPy for the same three tasks.

An ODE in one sentence

An ODE gives the rate of change of a quantity and asks for the quantity. Take a cooling object whose temperature difference from the room shrinks at a rate proportional to itself:

du/dt = -p ยท u, with u(0) = 1

Here u is the quantity, t is time and p is a constant. The exact answer is u(t) = exp(-pยทt), which makes this a good first test, because you can compare any numerical solution with the truth.

Solving it with DifferentialEquations.jl

Install with ] add DifferentialEquations (see Packages and Environments). Solving an ODE always follows three steps: define the function, build a problem, call solve.

JULIA
using DifferentialEquations

f(u, p, t) = -p * u          # the right-hand side: du/dt

u0    = 1.0                  # initial value
tspan = (0.0, 5.0)           # solve from t = 0 to t = 5
p     = 0.8                  # a parameter

prob = ODEProblem(f, u0, tspan, p)
sol  = solve(prob, Tsit5())  # Tsit5 is a good general-purpose method

The function f always takes three arguments in this order: the current state u, the parameters p and the time t. Even if f does not use t, the signature must keep it. The solution sol stores the time steps and values the solver chose:

JULIA
sol.t          # the times the solver visited
sol.u          # the state at each of those times
sol(2.5)       # the value at any time, by interpolation
sol[end]       # the final value, close to exp(-0.8 * 5)

Notice that you did not pick a step size. Tsit5 is an adaptive method: it takes big steps while the solution is smooth and small steps where it changes quickly, keeping the estimated error below tolerances you can set with reltol and abstol.

Solving an ODE in four steps

Write f(u, p, t) The rate of change.
Choose u0 and tspan Where you start and how long.
ODEProblem Bundle f, u0, tspan and p.
solve Pick a method such as Tsit5().
1 / 2

Systems of equations

Most interesting models have several variables. Then u is a vector, and for speed you write the function in place: it fills a pre-made array du instead of returning a new one. The name ends in ! by convention. Here is the classic predator and prey model, where x are prey and y are predators:

JULIA
function lotka!(du, u, p, t)
    ฮฑ, ฮฒ, ฮณ, ฮด = p
    x, y = u
    du[1] = ฮฑ * x - ฮฒ * x * y      # prey grow, and are eaten
    du[2] = -ฮณ * y + ฮด * x * y     # predators die, and grow by eating
end

u0   = [10.0, 5.0]
p    = (1.1, 0.4, 0.4, 0.1)
prob = ODEProblem(lotka!, u0, (0.0, 30.0), p)
sol  = solve(prob, Tsit5())

With Plots.jl (see Plotting), plot(sol) draws both populations over time. Because lotka! allocates nothing, the solver can run it millions of times quickly. If a solve is slow, or the step size collapses, your equations may be stiff, meaning some parts change very fast and some very slowly. Switch to a stiff method such as Rodas5() or Rosenbrock23(). Calling solve(prob) with no method lets the package choose a default for you.

The same ODEProblem, solve pattern covers many other problem types in the package: stochastic equations, delay equations, boundary value problems and more. The workflow stays the same.

Optimization with constraints: JuMP

JuMP is a modeling language for mathematical optimization embedded in Julia. You describe variables, constraints and an objective in near-mathematical notation, and pass the model to a solver. Suppose a workshop makes two products, a and b. Each unit of a uses 2 hours of labor and 1 kg of material, and each unit of b uses 1 hour and 1 kg. There are 100 hours and 80 kg available, and at most 40 units of a can be sold. A unit of a earns 40, and a unit of b earns 30. How many of each should be made?

JULIA
using JuMP, HiGHS

model = Model(HiGHS.Optimizer)

@variable(model, a >= 0)
@variable(model, b >= 0)

@constraint(model, labor,    2a + b <= 100)
@constraint(model, material, a + b <= 80)
@constraint(model, demand,   a <= 40)

@objective(model, Max, 40a + 30b)

optimize!(model)

termination_status(model)    # OPTIMAL
value(a), value(b)           # (20.0, 60.0)
objective_value(model)       # 2600.0

The macros @variable, @constraint and @objective turn the readable formulas into the matrices a solver needs. HiGHS is an open-source solver for linear and mixed-integer problems, and other solvers (Ipopt for nonlinear, SCS and Clarabel for conic) plug in the same way by changing the constructor in Model(...). You can name constraints, as above, and ask about them later: dual(labor) is the extra profit from one more hour of labor, the "shadow price". For integer decisions, declare a variable as @variable(model, n >= 0, Int).

Unconstrained minimization: Optim.jl

When you simply want the input that makes a function smallest, with no constraints, use Optim.jl. Fitting a curve to data is the typical case: define the squared error as a function of the parameters and minimize it.

JULIA
using Optim

f(x) = (x[1] - 3)^2 + (x[2] + 1)^2 + 5     # lowest at (3, -1), where it equals 5

res = optimize(f, [0.0, 0.0], LBFGS())

Optim.minimizer(res)    # approximately [3.0, -1.0]
Optim.minimum(res)      # approximately 5.0
Optim.converged(res)    # true

The vector [0.0, 0.0] is the starting guess. LBFGS() uses gradients, which Optim estimates by finite differences unless you pass autodiff = :forward to get exact gradients through automatic differentiation. Without a method, optimize(f, x0) uses NelderMead(), which needs no gradients and is slower but robust. For a single variable, optimize(f, lower, upper) searches an interval. Optimizers find a local minimum near the start, so for functions with many valleys, try several starting points.

What a solver does inside: Euler in Python

An ODE solver turns a continuous problem into small steps. The simplest scheme, the Euler method, moves along the slope: u_next = u + h ยท f(u, t). This stand-in applies it to the cooling problem and compares with the exact answer at t = 5. As you shrink the step h, the error shrinks too.

Python Euler's method against the exact answer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each tenfold decrease of h cuts the error about tenfold, which is what "first-order" means. Methods like Tsit5 use several slope evaluations per step to be far more accurate for the same effort, and they choose h automatically. The next example shows both ideas with SciPy's adaptive solver, plus the other two tasks of this lesson. Remember that it is only a stand-in: the names differ, and solve_ivp plays the part of solve.

Python The same three tasks with SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the argument order of an ODE function f in DifferentialEquations.jl?

    The state comes first, then the parameters, then the time. The function must accept all three even when it ignores some.
  2. 2. Why do ODE functions for systems usually have a name like lotka!(du, u, p, t)?

    The ! signals mutation. Writing the derivatives into du instead of returning a fresh array makes the solver faster.
  3. 3. In the JuMP model, what does @objective(model, Max, 40a + 30b) do?

    @objective sets the quantity to maximize or minimize; constraints are declared with @constraint.
  4. 4. When is Optim.jl a better fit than JuMP?

    Optim.jl minimizes ordinary Julia functions freely. JuMP is built for problems stated with variables and constraints.
  5. 5. In the Euler stand-in, what happens to the error when the step size is divided by ten?

    Euler is a first-order method, so the error is proportional to the step size.

Summary

  • An ODE problem is built from f(u, p, t), an initial value u0, a time span and parameters, then solved with solve(prob, Tsit5()).
  • For systems, write the function in place as f!(du, u, p, t); for stiff problems choose a method such as Rodas5().
  • JuMP describes optimization with constraints using @variable, @constraint and @objective, and hands the model to a solver such as HiGHS.
  • Optim.jl minimizes a plain function from a starting guess with optimize, using LBFGS() or NelderMead().
  • Solvers work by repeated small steps; the Euler stand-in shows why smaller steps give smaller errors.

Next: draw random numbers and summarize data with Random Numbers and Statistics.