Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 38 ยท Optimization

Minimizing Functions of Many Variables

Use scipy.optimize.minimize on functions of many variables. Compare BFGS, Nelder-Mead and L-BFGS-B, supply gradients, add bounds and callbacks, read the result.

Most interesting optimization problems have more than one unknown. You tune the weights of a model, choose the dimensions of a box, or search for the parameters that make a simulation match measurements. In each case the thing you minimize is a function that takes a vector x and returns a single number: a cost, a loss, an error.

scipy.optimize.minimize is the single entry point for this. You pass the function and a starting point, optionally name a method, and get back an OptimizeResult. This lesson shows how to choose a method, why gradients matter, how to add simple bounds, how to watch the search with a callback, and why a "successful" run can still land in the wrong valley. For one-variable problems, see Scalar Minimization and Root Finding first.

The Rosenbrock test function

Optimizers are traditionally tested on the Rosenbrock function, f(x, y) = 100 (y - xยฒ)ยฒ + (1 - x)ยฒ. Its minimum is exactly zero at (1, 1), at the end of a long, curved, narrow valley. Reaching the valley floor is easy; creeping along it is hard, which makes it a good stress test. SciPy ships it ready-made as rosen (with its gradient rosen_der) in scipy.optimize.

flowchart LR
    F["objective f(x)<br/>takes a vector,<br/>returns a number"] --> M["minimize(f, x0, method=...)"]
    X0["starting point x0"] --> M
    O["optional: jac, bounds,<br/>constraints, callback, options"] --> M
    M --> R["OptimizeResult<br/>x, fun, success,<br/>nit, nfev, message"]

The inputs and output of minimize: the function gets a one-dimensional array and returns a scalar.

Python Your first minimize call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The important part of the result object:

Field Meaning
x the best point found (a NumPy array)
fun the objective value at x
success, message whether the solver met its stopping rule, and in words
nit iterations
nfev, njev number of objective and gradient evaluations

When you do not pass method, SciPy chooses for you: BFGS if there are no bounds or constraints, L-BFGS-B if there are bounds, and SLSQP if there are constraints. The function you minimize must accept one array argument; extra fixed values go through args=(...).

Choosing a method

Methods fall into two groups. Derivative-free methods only call f. Gradient-based methods also use the slope of f, which tells them which way is downhill, and are usually much faster.

Method Uses gradient Bounds Best for
Nelder-Mead no yes (recent SciPy) noisy or non-smooth functions, few variables
Powell no yes smooth-ish functions without derivatives
BFGS yes (estimated if missing) no smooth problems up to a few hundred variables
L-BFGS-B yes yes large problems, simple box bounds
CG, Newton-CG yes no memory-lean or Hessian-aware problems

If you leave out jac, SciPy estimates the gradient with finite differences: it nudges each variable and watches f. That costs n extra calls per gradient. Supplying the true gradient saves those calls and gives more accurate steps. Compare:

Python Methods and gradients compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every method finds (1, 1). The cost differs: Nelder-Mead needs the most function calls, BFGS with finite differences spends many calls on slope estimates, and BFGS with an exact jac needs far fewer. (Exact counts can vary a little between platforms; the ordering is what matters.)

Writing and checking a gradient

For f(x, y) = (x - 3)ยฒ + 2(y + 1)ยฒ + x*y the partial derivatives are 2(x - 3) + y and 4(y + 1) + x. A wrong gradient is a nasty bug, because the optimizer will wander or stop early. scipy.optimize.check_grad compares your gradient with a finite-difference estimate, so run it once on a test point.

Python Supplying a gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because this function is quadratic, the exact minimum is a linear solve, and both runs reproduce it. That is a neat check to keep in mind: whenever you can solve a simplified version in closed form, compare.

If you cannot write a gradient by hand, you have three options: let SciPy use finite differences, use jac='3-point' for a more accurate (but costlier) estimate, or use an automatic-differentiation library such as JAX outside SciPy.

Watching the search with a callback

callback is a function called after each iteration with the current point. It is perfect for logging, early stopping (raise StopIteration), or recording the path. Here we draw the paths of two methods over the contour lines of Rosenbrock:

Python Optimizer paths on the Rosenbrock valley

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The gradient method takes noticeably fewer, longer steps and bends around the valley, while Nelder-Mead shuffles a small triangle of points (a simplex) along it. Note that the callback receives the current point xk only; the exact signature differs slightly by method, so keep it to one argument and copy the array if you store it, since solvers may reuse the buffer.

Adding bounds

If a variable has a physical range (a probability between 0 and 1, a length that must be positive), pass bounds as a list of (low, high) pairs, one per variable, with None meaning unbounded. L-BFGS-B is the workhorse here, and SciPy picks it automatically when you pass bounds without naming a method.

Python Bounds with L-BFGS-B

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the true minimum (1, 1) lies outside the box (x โ‰ค 0.5), the best feasible point sits on the boundary at x = 0.5. Optimizers handle this fine; the result simply reports the constrained optimum. More general constraints, such as x + y โ‰ค 1, come in Constraints, Bounds and Global Optimization.

Local minima and restarts

Every method above is local: it walks downhill from x0 and stops in the first valley it finds. If the landscape has several valleys, the starting point decides the answer. The function below is a bumpy surface with many dips.

Python Restarts from random points

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Thirty starts reveal several local minima with different depths; only the best is the global minimum of this region. A single run would have given you one of them at random. The pragmatic recipe is multi-start: run from many points and keep the lowest fun. For rugged problems, SciPy also offers dedicated global optimizers, covered in the constrained-and-global lesson.

Scale your variables so they are of similar size. An optimizer struggles when one parameter lives near 0.001 and another near 1,000, because a single step size cannot suit both. Rescale to comparable units before calling minimize.

Reading the result honestly

A good habit after any run:

  1. Check res.success and read res.message. A message about "maximum iterations" or "precision loss" means do not trust the answer.
  2. Evaluate the objective at res.x yourself and compare with res.fun.
  3. Try a second starting point; if the answers differ, you have multiple minima.
  4. Lower tol, or raise options={"maxiter": ...}, if a run stops early on a hard problem.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What kind of function does scipy.optimize.minimize expect as its first argument?

    The objective takes a single 1-D array of variables and returns a scalar. Extra fixed values are passed through the args parameter.
  2. 2. What happens if you do not pass jac to a method like BFGS?

    SciPy estimates the gradient numerically by nudging each variable. This works but needs extra evaluations of f, so an exact gradient is faster and more accurate.
  3. 3. You run minimize from one starting point and get success=True. Which statement is safest?

    success only means the solver stopped properly. These methods are local, so the starting point decides which valley you reach.
  4. 4. Which method is the natural choice for a large smooth problem with simple lower and upper bounds on variables?

    L-BFGS-B supports box bounds, uses gradients and has modest memory needs, so it scales to many variables.

Summary

  • minimize(f, x0, method=..., jac=..., bounds=..., callback=...) minimizes a function of a vector; the result holds x, fun, success, message, nit and nfev.
  • Derivative-free methods (Nelder-Mead, Powell) are robust for rough functions; gradient methods (BFGS, L-BFGS-B, CG) are faster on smooth ones.
  • Supplying an exact jac saves function calls; verify it with check_grad.
  • Use bounds with L-BFGS-B for simple box limits; the optimum may sit on the boundary.
  • A callback lets you record or stop the search.
  • All of these find local minima: use multi-start or a global optimizer when the landscape has many valleys.

Next up: Constraints, Bounds and Global Optimization.