Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 38 ยท Optimization

Constraints, Bounds and Global Optimization

Add equality and inequality constraints with SLSQP and trust-constr, then escape local minima using differential_evolution, dual_annealing, basinhopping and shgo.

Real decisions come with rules. A rectangle must have a fixed perimeter, a portfolio's weights must add up to one, a budget cannot be exceeded, a dose must stay below a safe limit. Mathematically these are constraints, and they change the problem: the best point is no longer where the function is lowest, but where it is lowest among the allowed points.

A second difficulty is the shape of the function itself. The local solvers from Minimizing Functions of Many Variables walk downhill and stop at the first valley. If the surface has hundreds of dips, you need global optimizers, which sample the whole region. This lesson covers both: constrained local solvers (SLSQP, trust-constr) and four global ones (differential_evolution, dual_annealing, basinhopping, shgo).

flowchart TD
    Q["What does the problem look like?"] --> A["Smooth, with constraints,<br/>one good starting point"]
    Q --> B["Many local minima,<br/>only box bounds"]
    A --> A1["minimize with SLSQP<br/>or trust-constr"]
    B --> B1["differential_evolution<br/>dual_annealing<br/>shgo, basinhopping"]
    B1 --> P["optionally polish with<br/>a local solver"]

Local solvers handle constraints near a good starting point; global solvers explore the whole box.

Equality and inequality constraints with SLSQP

SLSQP (sequential least-squares quadratic programming) handles bounds, equality constraints and inequality constraints for smooth problems. Constraints are described as dictionaries with a type and a fun:

  • {"type": "eq", "fun": g} means g(x) = 0.
  • {"type": "ineq", "fun": g} means g(x) >= 0.

The sign convention for ineq trips up many people: the function must be non-negative when the point is allowed. To say x + y <= 10, write 10 - x - y.

Here are two classic problems. First, the rectangle of largest area with a perimeter of 20. SciPy minimizes, so we minimize the negative area. Second, the point of the unit disc closest to (3, 3).

Python SLSQP with eq and ineq constraints

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rectangle turns out to be a square, as geometry predicts. The disc problem shows the usual behavior of inequality constraints: because (3, 3) is outside the disc, the best allowed point lies on the boundary, on the line from the origin toward (3, 3). The constraint is then called active. When the unconstrained optimum is inside the allowed region, the constraint is inactive and has no effect.

Start from a point that satisfies the constraints if you can. SLSQP will cope with infeasible starts, but a feasible one gives it an easier time and fewer chances to stall.

trust-constr and constraint objects

trust-constr is the more modern and more robust constrained solver. It prefers constraint objects, which carry the bounds on the constraint value itself (lower and upper), so you can write ranges naturally:

  • LinearConstraint(A, lb, ub) means lb <= A @ x <= ub.
  • NonlinearConstraint(fun, lb, ub) means lb <= fun(x) <= ub.
  • Bounds(lb, ub) limits the variables themselves.

Use np.inf and -np.inf for open ends. Constraint objects also work with SLSQP in current SciPy.

Python Constraint objects with trust-constr

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unconstrained minimum of Rosenbrock is (1, 1), which violates x + 2y <= 1. The solver slides along that line until it can go no lower: the linear constraint is active, the curved one has slack. Two practical notes: trust-constr accepts hess and sparse Jacobians for large problems, and it is slower per iteration than SLSQP on small ones. If one fails to converge, try the other.

Why global optimizers?

Local solvers cannot see beyond the valley they start in. Global optimizers take a different approach: they work with many candidate points at once, or they accept occasional uphill moves, so they can escape. They need finite bounds on every variable so that "the whole space" is a well-defined box, and they usually do not need gradients. The price is many more function evaluations.

The classic stress test is the Rastrigin function, a bowl covered in regularly spaced dips. Its global minimum is zero at the origin, but it has a huge number of local minima around it. We write it ourselves with NumPy:

Python A local solver gets trapped; differential evolution does not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The local solver ends in a nearby dip, with a value well above zero. differential_evolution keeps a population of candidate points (popsize times the number of variables), repeatedly combines and mutates them, and keeps improvements. With the default population and this seed it can still settle in a near-miss dip; a larger population searches more widely and reaches the true minimum. That is typical of global methods: they are stochastic, so use a seed for reproducibility and raise popsize or maxiter when you doubt the result. By default differential_evolution also finishes with a short local "polish" to sharpen the answer.

Four global optimizers compared

SciPy provides several global strategies, each with a different personality:

Function Idea Good for
differential_evolution evolve a population of points general-purpose; also supports constraints
dual_annealing simulated annealing plus local search rugged, continuous landscapes
basinhopping random jumps, each followed by a local minimization molecular and "funnel-shaped" landscapes
shgo builds a topology of the function and finds all local minima low dimensions; Lipschitz-smooth problems, can return many minima

Himmelblau's function, (xยฒ + y - 11)ยฒ + (x + yยฒ - 7)ยฒ, has four equal minima, all with value zero. Any of them is a correct answer, so we test whether each optimizer lands on one.

Python Four global optimizers on Himmelblau

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All four reach a true minimum. Which one of the four equivalent points they report may differ, and with other seeds it will. Notice that basinhopping takes a starting point and a minimizer_kwargs dictionary that is passed to minimize, while the others take bounds only. shgo is the odd one out: it is deterministic, and its result carries xl and funl, the full lists of local minima it found, which is useful when you want all good solutions instead of one.

Practical advice

  • Prefer a local solver when you can. If the problem is smooth and you have a good guess, L-BFGS-B or SLSQP is hundreds of times cheaper than any global method.
  • Combine them. Run a global optimizer with loose tolerance to find the right region, then polish with a local solver. differential_evolution and dual_annealing already include a polishing step.
  • Bound every variable tightly. The global search space grows exponentially with dimension, so a smaller box helps enormously.
  • Budget the cost. Each of these can make tens of thousands of function calls. For slow objectives, lower maxiter and popsize, or run on a coarser model first.
  • Repeat with different seeds. If two seeds give different answers, the landscape is hard and the best value found is only an upper bound on the true minimum.

For fitting a model to data, a specialized tool is better than a generic minimizer; that comes up in Curve Fitting and Least Squares.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a SLSQP constraint {'type': 'ineq', 'fun': g}, which condition is enforced?

    SciPy's convention is that an inequality constraint holds when its function is non-negative. To express x + y <= 10, use 10 - x - y.
  2. 2. Why do global optimizers such as differential_evolution require finite bounds on every variable?

    These methods sample or move across the whole region, so the region must be a finite box. Gradients are not required.
  3. 3. A constrained optimum lies on the boundary of the allowed region. What do we call that constraint?

    A constraint that holds with equality at the solution is active. It is the reason the answer differs from the unconstrained optimum.
  4. 4. A differential_evolution run returns a value slightly above the known global minimum. Which reaction is most sensible?

    Global optimizers are stochastic and can settle in a near-miss dip. More population, more iterations, other seeds and a local polish improve the answer.

Summary

  • Constraints restrict the allowed points: SLSQP takes dictionaries ("eq" means g(x) = 0, "ineq" means g(x) >= 0), and trust-constr takes LinearConstraint, NonlinearConstraint and Bounds objects.
  • Active constraints put the optimum on the boundary; inactive ones do nothing.
  • Local solvers stop in the first valley; global optimizers (differential_evolution, dual_annealing, basinhopping, shgo) search the whole bounded region at much higher cost.
  • Global methods are stochastic: set seed, increase the population or iterations when in doubt, and polish locally.
  • Use a local solver whenever a good starting guess exists.

Next up: Curve Fitting and Least Squares.