Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 45 ยท Numerical Methods

Solving Equations and Optimization

Solve nonlinear equations and minimize functions in MATLAB and Octave with fzero, fsolve, fminsearch, fminunc, fmincon and linprog, with scipy.optimize stand-ins.

Many problems come down to two questions. Where does this function equal zero? And where is it smallest or largest? Few real equations have a formula for the answer, so MATLAB and Octave offer numerical solvers that start from a guess and improve it step by step: fzero for one equation in one unknown, fsolve for several, and fminsearch, fminunc and fmincon for minimization.

All of these take a function handle (see Anonymous Functions and Function Handles). MATLAB and Octave are not available in your browser here, so the MATLAB code is read-only; each example has a runnable Python/SciPy stand-in using scipy.optimize, which offers the same algorithms under different names. fzero and fminsearch are in core MATLAB and core Octave; fsolve is in core Octave and in MATLAB's Optimization Toolbox; fminunc, fmincon and linprog belong to the Optimization Toolbox in MATLAB.

Two families of problems

Root finding Find x where f(x) = 0.
Optimization Find x where f(x) is smallest.

Maximizing f is minimizing -f.

1 / 2

One equation: fzero

fzero(f, x0) finds a point where f crosses zero. With a scalar x0 it searches outward for a sign change; with a two-element bracket [a b] it requires f(a) and f(b) to have opposite signs, which guarantees a root and makes the method very reliable.

MATLAB
f = @(x) cos(x) - x;           % solve cos(x) = x
x = fzero(f, 0.5)              % 0.7391
x = fzero(f, [0 1])            % same, with a bracket
[x, fval, flag] = fzero(f, [0 1])   % fval ~ 0, flag = 1 means success

Move everything to one side so the equation reads f(x) = 0. To solve exp(-x) = x^2, write f = @(x) exp(-x) - x.^2. A function can have several roots; fzero finds only one, the one nearest your start or inside your bracket. Plot the function first (see 2-D Plots) to see how many sign changes there are and where.

Python Solve cos(x) = x

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Several equations: fsolve

For a system, write a function that takes a vector of unknowns and returns a vector of residuals (each equation moved to the form ... = 0). fsolve needs a starting vector.

MATLAB
% x^2 + y^2 = 4   and   y = x
F  = @(v) [v(1)^2 + v(2)^2 - 4;
           v(2) - v(1)];
v0 = [1; 1];
v  = fsolve(F, v0)                       % [1.4142; 1.4142]

Nonlinear systems can have several solutions (here also [-1.4142; -1.4142]). Which one you get depends on the starting point. Check the exit flag (the third output) and verify the residual F(v) is near zero. If the equations are linear, skip fsolve and use the backslash operator from Linear Algebra Essentials.

Python A circle meets a line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can see that different starting points reach different solutions. Always choose starting values from physical sense or a plot.

Minimization without derivatives: fminsearch

fminsearch(f, x0) minimizes a function of one or more variables using the Nelder-Mead simplex method, which only evaluates f and never needs a gradient. The function takes one vector argument, even when it has several variables.

MATLAB
f = @(p) (p(1) - 3)^2 + (p(2) + 1)^2 + 5;      % bowl with minimum at (3, -1)
[p, fmin] = fminsearch(f, [0 0])                % p ~ [3 -1], fmin ~ 5

g = @(x) x.^4 - 3*x.^2 + x;
[xm, gm] = fminsearch(g, 0)                     % x = -1.30, g = -3.51 (deeper valley)
[xm2, gm2] = fminsearch(g, 2)                   % x = 1.13, g = -1.07 (shallower valley)

fminbnd(g, a, b) is the one-dimensional version restricted to an interval. For maximizing, minimize the negative. Because fminsearch stops when the simplex is small, its answers are typically accurate to a few digits, not to full machine precision.

Here is an example worth running: g has two valleys, so the result depends on where you start. That is the key difficulty of optimization: algorithms find a local minimum, not necessarily the global one.

Python Two valleys, two answers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Starting at -2 or at 0 reaches the deeper left valley (the global minimum, about -3.51); starting at 2 lands in the shallower right-hand one (about -1.07), which is only a local minimum. When a problem may have several minima, run the solver from several starting points and keep the best.

Smooth and constrained problems: fminunc, fmincon, linprog

When the function is smooth, gradient-based solvers converge faster and more precisely than fminsearch. They belong to MATLAB's Optimization Toolbox (Octave has fminunc in core, and sqp for constraints).

MATLAB
% unconstrained, smooth
p = fminunc(f, [0 0]);

% with constraints:  minimize f  subject to  p(1) + p(2) <= 1  and  p >= 0
A = [1 1];  b = 1;                    % linear inequality  A*p <= b
lb = [0 0];                           % lower bounds
p = fmincon(f, [0.5 0.5], A, b, [], [], lb, []);

% linear programming:  minimize c'*x  subject to  A*x <= b,  x >= 0
c = [-3; -5];                         % maximize 3x + 5y  <=>  minimize -3x - 5y
A = [1 0; 0 2; 3 2];  b = [4; 12; 18];
x = linprog(c, A, b, [], [], [0; 0])  % [2; 6]

linprog solves problems where both the objective and constraints are linear, which is a much easier class than general nonlinear problems and has a guaranteed global answer. The classic production-planning example above gives x = 2, y = 6, with a profit of 3*2 + 5*6 = 36.

Python A constrained problem and a linear program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unconstrained minimum at (3, -1) violates the constraints, so the constrained answer sits on the boundary, at (1, 0).

Choosing starting points and tolerances

  • Plot first. For one variable a plot shows roots and valleys at a glance; for two variables use contour (see 3-D Plots, Surfaces and Contours).
  • Try several starts if the problem might have more than one answer, and compare the results.
  • Scale your variables. If one unknown is near 0.001 and another near 1000, solvers struggle; rescale so both are of order 1.
  • Read the exit flag, the third or fourth output. A result without a check on convergence is just a guess.
  • Tolerances are set with optimset (or optimoptions in the toolbox): opts = optimset('TolX', 1e-10, 'Display', 'iter'); fzero(f, [0 1], opts).
  • Pass extra parameters through a closure: k = 2; f = @(x) exp(-k*x) - x; captures k without globals.
  • Check the answer. Substitute it back: f(x) should be tiny, and the gradient near zero at a minimum.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which form must an equation have before you pass it to fzero?

    fzero finds zeros of a function, so move all terms to one side: solve cos(x) = x as cos(x) - x = 0.
  2. 2. What is required of a bracket [a b] given to fzero?

    A continuous function with opposite signs at both ends must cross zero in between, which guarantees a root.
  3. 3. How do you find the maximum of f with fminsearch?

    Maximizing f is the same as minimizing -f; the location is identical, and the maximum value is the negative of the minimum found.
  4. 4. Why can fminsearch return different answers from different starting points?

    Numerical minimizers walk downhill from the start and stop at the first valley they reach.
  5. 5. What is special about linprog compared with fminsearch?

    Linear programs are a well-behaved class: no local-versus-global ambiguity, and dedicated algorithms solve them reliably.

Summary

  • Rewrite equations as f(x) = 0 and use fzero (with a bracket for safety) or fsolve for systems.
  • fminsearch minimizes without gradients; fminbnd works on an interval; fminunc and fmincon use gradients and constraints; linprog solves linear programs.
  • Solvers find local answers from a starting point: plot, try several starts, scale variables and check exit flags.
  • Maximize by minimizing the negative; pass parameters with anonymous-function closures.
  • scipy.optimize offers matching algorithms (brentq, fsolve, minimize, linprog) for the stand-ins here.

Next, integrate, differentiate and simulate change over time in Integration, Differentiation and ODEs.