Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 39 · Optimization in Several Variables

Lagrange Multipliers

Maximize or minimize f subject to a constraint g = c using Lagrange multipliers: geometric idea, worked examples in two and three variables, SymPy code.

Real problems come with limits. You want the biggest box, but you only have so much cardboard. You want the cheapest mix, but it must contain 2 kg of protein. Such problems ask for the maximum or minimum of a function f subject to a constraint g(x, y) = c.

Sometimes you can solve the constraint for one variable and substitute. When that is messy, Lagrange multipliers give a clean, symmetric method that works with any number of variables.

The geometric idea

Picture the level curves of f as contour lines, and the constraint as a path drawn on the map. Walk along the path and watch the contour value change. At a maximum along the path, you are touching a contour line without crossing it. So the path and the contour line are tangent there.

The gradient of f is perpendicular to its contour line, and the gradient of g is perpendicular to the path. If the two curves are tangent, those gradients are parallel:

Text
∇f = λ ∇g

The number λ (lambda) is the Lagrange multiplier. Together with the constraint g = c, this gives the system to solve.

The method

To optimize f(x, y) subject to g(x, y) = c:

  1. Write fx = λ gx, fy = λ gy, and g = c. That is three equations for three unknowns x, y, λ.
  2. Solve the system. Each solution (x, y) is a candidate.
  3. Evaluate f at every candidate. The largest is the constrained maximum, the smallest the constrained minimum (when the constraint curve is closed and bounded, so extremes exist).

The same recipe works for three variables: fx = λ gx, fy = λ gy, fz = λ gz, g = c.

Check that ∇g is not the zero vector at your candidates. If ∇g = 0 somewhere on the constraint, the method can miss that point, so examine it separately.

Example 1: a circle

Maximize and minimize f(x, y) = xy on the circle x² + y² = 8.

Here ∇f = (y, x) and ∇g = (2x, 2y). The equations are

Text
y = 2λx
x = 2λy
x² + y² = 8

Multiply the first by y and the second by x: y² = 2λxy = x². So y = ±x.

  • If y = x: 2x² = 8, so x = ±2. Points (2, 2) and (-2, -2), with f = 4.
  • If y = -x: points (2, -2) and (-2, 2), with f = -4.

The maximum of xy on this circle is 4; the minimum is -4.

Python Solve the Lagrange system with SymPy

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What does lambda mean?

The multiplier measures sensitivity. If you relax the constraint from g = c to g = c + Δc, the optimal value of f changes by about λ Δc. In economics this is the "shadow price": how much extra output one more unit of the limited resource buys. In Example 1, increasing the radius squared slightly raises the best value of xy, and λ tells you by how much per unit.

Example 2: the best box

A closed box with sides x, y, z has surface area 2(xy + yz + zx) = 24 square units. Which dimensions give the largest volume?

Maximize V = xyz subject to 2(xy + yz + zx) = 24, that is xy + yz + zx = 12.

Text
yz = λ (y + z)
xz = λ (x + z)
xy = λ (x + y)
xy + yz + zx = 12

Multiply the equations by x, y, z respectively:

Text
xyz = λ x(y + z)
xyz = λ y(x + z)
xyz = λ z(x + y)

Since xyz > 0 for a real box, λ ≠ 0, so xy + xz = yx + yz = zx + zy. From the first two, xz = yz, so x = y. From the last two, yx = zx, so y = z. All sides are equal and 3x² = 12, giving x = 2. The best box is a cube with side 2 and volume 8.

Python Numerical check: random boxes never beat the cube

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random search is only a check. The multiplier solution is the exact argument.

Two constraints

With two constraints g = c and h = d in three variables, you add a second multiplier:

Text
∇f = λ ∇g + μ ∇h

and solve it together with both constraint equations. The geometry is a curve (the intersection of two surfaces), and ∇f must lie in the plane spanned by the two constraint gradients.

When to use something else

If the constraint is easy to parametrize (a circle as (cos t, sin t)), substitution can be quicker, as in absolute extrema. Multipliers shine when the constraint is awkward to solve for a variable or when there are several variables.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. At a constrained extremum of f subject to g = c, what is the relationship between the gradients?

    At the optimum the constraint curve is tangent to a level curve of f, so both gradients are normal to the same direction and are parallel.
  2. 2. In Example 1 (maximize xy on x² + y² = 8), what is the maximum value?

    The candidates are (±2, ±2). Points with equal signs give xy = 4 and points with opposite signs give -4.
  3. 3. How many unknowns are in the Lagrange system for two variables and one constraint?

    There are three equations (fx = λgx, fy = λgy, and g = c) in the three unknowns x, y and λ.
  4. 4. Roughly, what does λ tell you?

    λ is the sensitivity of the optimal value to the constraint level, often called the shadow price in economics.

Summary

  • To optimize f subject to g = c, solve ∇f = λ∇g together with g = c.
  • The geometry: the constraint is tangent to a level curve of f at the optimum.
  • Evaluate f at all solutions and compare to pick the maximum and minimum.
  • λ measures how the optimum responds to changes in c.
  • Two constraints use ∇f = λ∇g + μ∇h.

Next: Least Squares and Gradient Descent