Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 68 ยท Multivariable Calculus

Optimization in Several Variables

Find minima, maxima and saddle points of functions of several variables with the Hessian test, use Lagrange multipliers for constraints, and run gradient descent.

Engineers optimize all the time: the cheapest material for a tank of fixed volume, the angle that maximizes range, the parameters that minimize the error of a model. With one variable you set the derivative to zero. With several variables the idea is the same, but there are new ways for a point to be "flat" without being a peak or a valley, and constraints add a new twist.

This lesson gives you three tools. First, the gradient test finds candidate points. Second, the Hessian test classifies them as minima, maxima or saddles. Third, Lagrange multipliers handle problems with constraints. We finish with gradient descent, the numeric method that works when equations cannot be solved by hand. You should be comfortable with Partial Derivatives and Gradient and Directional Derivatives.

flowchart TD
    A["Solve grad f = 0<br/>critical points"] --> B["Compute the Hessian<br/>at each point"]
    B --> C{"D = f_xx f_yy - f_xy^2"}
    C -->|"D > 0 and f_xx > 0"| M["Local minimum"]
    C -->|"D > 0 and f_xx < 0"| X["Local maximum"]
    C -->|"D < 0"| S["Saddle point"]
    C -->|"D = 0"| U["Test inconclusive"]

A two-variable second-derivative test: find where the gradient vanishes, then read the sign of D.

Critical points

At a local minimum or maximum of a smooth function, the surface is level in every direction, so every directional derivative is zero. That means the whole gradient vanishes:

grad f = 0, that is f_x = 0 and f_y = 0 at the same time.

Points where this happens are critical points. They are only candidates: a critical point might be a minimum, a maximum, or a saddle, where the surface rises in one direction and falls in another (like a mountain pass). Solving grad f = 0 is a system of equations, and sympy.solve can often do it exactly.

The Hessian and the second-derivative test

To classify a critical point, look at the curvature. The Hessian matrix collects all second partials:

Text
H = [ f_xx  f_xy ]
    [ f_yx  f_yy ]

For two variables define the discriminant D = f_xx * f_yy - f_xy**2 (the determinant of H). At a critical point:

  • D > 0 and f_xx > 0: local minimum (curves up in all directions).
  • D > 0 and f_xx < 0: local maximum.
  • D < 0: saddle point.
  • D = 0: the test cannot decide; look at the function more closely.

Another way to say the same thing uses the eigenvalues of H: all positive means a minimum, all negative a maximum, and mixed signs a saddle. That version works for any number of variables.

Take f(x, y) = x**3 - 3*x*y + y**3. Then f_x = 3x**2 - 3y and f_y = 3y**2 - 3x.

Python Find and classify critical points

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The origin has D = -9, so it is a saddle, and the point (1, 1) has D = 27 and f_xx = 6, so it is a local minimum with value -1. The eigenvalues tell the same story: both positive at (1, 1), with one positive and one negative at the origin.

Seeing a saddle

A saddle is easier to believe once you have seen one. Plot the surface near the origin and a slice along two different directions.

Python A saddle and a minimum on the same surface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Along y = x the value f is negative near the origin (it dips below f(0, 0) = 0), and along y = -x it is positive. The origin is therefore neither a minimum nor a maximum: the contour lines cross there in an X shape, the signature of a saddle.

Constrained optimization: Lagrange multipliers

Often you cannot choose the variables freely. You want the largest area of a fence with a fixed length, the smallest cost with a fixed output, or the shortest distance to a surface. A constraint g(x, y) = c restricts the search to a curve (or surface).

At the best point on the constraint curve, a level curve of f just touches the constraint curve without crossing it. Touching curves share a tangent, so their normals are parallel. The normals are the gradients, which gives the Lagrange condition:

grad f = lambda * grad g, together with the constraint g = c.

The unknown number lambda is the Lagrange multiplier. For n variables you get n + 1 equations in n + 1 unknowns. A pleasant bonus: lambda measures how fast the optimal value changes if you relax the constraint, which engineers read as a "shadow price".

As an example, find the point of the plane x + 2y + 3z = 14 that is closest to the origin. Minimizing the distance is the same as minimizing the squared distance f = x**2 + y**2 + z**2, which avoids the square root.

Python Lagrange multipliers with sympy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The closest point is (1, 2, 3) with squared distance 14. The second problem returns four points: the two with x and y of the same sign give the maximum xy = 4, and the two with opposite signs give the minimum -4. Lagrange's equations find every extreme point, so you must compare the values to decide which is which.

Lagrange multipliers find candidates, not winners. After solving, always evaluate f at every candidate and compare. Also check the case where grad g = 0, because the method divides that out.

Gradient descent

Sometimes grad f = 0 cannot be solved by hand. Gradient descent is the numeric alternative: start anywhere and repeatedly step downhill, since -grad f is the direction of steepest descent:

p_next = p - alpha * grad f(p).

The positive number alpha is the learning rate (step size). Choose it too small and you crawl; too big and you overshoot and diverge. For a quadratic bowl whose largest curvature (Hessian eigenvalue) is L, the iteration is stable when alpha < 2 / L.

Try it on f(x, y) = (x - 1)**2 + 10*(y + 2)**2. Its minimum is at (1, -2), and its Hessian is diag(2, 20), so the stability limit is alpha < 2/20 = 0.1.

Python Gradient descent and the learning rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With alpha = 0.01 the iteration is still far from the answer after 60 steps, with 0.05 and 0.09 it has converged, and with 0.11 (just beyond the limit 0.1) the y coordinate flips sign and grows every step, so the method diverges. For alpha = 0.05 the y error is multiplied by 1 - 20(0.05) = 0 on the first step, so the path drops straight to y = -2 and then crawls along the flat direction toward x = 1: the bowl is much steeper in y than in x, and that imbalance is why one learning rate suits one direction better than the other. You will tune and extend this method (momentum, fitting data) in Project: Gradient Descent from Calculus.

Choosing a method

  • Small problems with formulas: solve grad f = 0 with sympy and apply the Hessian test.
  • Constraints with equalities: Lagrange multipliers, or substitute the constraint to remove a variable.
  • Many variables or no closed form: gradient descent, or better methods from scipy.optimize that use the same gradient.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is true of every local minimum or maximum of a smooth function of two variables?

    At a local extreme, the function cannot increase or decrease to first order in any direction, so f_x = f_y = 0. This is a necessary condition, not a sufficient one.
  2. 2. At a critical point D = f_xx f_yy - f_xy^2 equals -4. What is the point?

    A negative discriminant means the surface curves up in one direction and down in another, which is a saddle.
  3. 3. What does the Lagrange condition grad f = lambda * grad g express?

    At the constrained optimum, the gradients (normals) of f and g point along the same line, so one is a multiple of the other.
  4. 4. For f = (x-1)^2 + 10(y+2)^2 the Hessian is diag(2, 20). Which learning rate makes gradient descent diverge?

    Stability needs alpha < 2 / 20 = 0.1. At alpha = 0.11 the update for y multiplies the error by 1 - 20(0.11) = -1.2, so it grows each step.
  5. 5. After solving the Lagrange equations you get several candidate points. What must you do next?

    The equations find every constrained extreme point, minima and maxima alike. Comparing the values of f tells you which is which.

Summary

  • Critical points satisfy grad f = 0; they can be minima, maxima or saddles.
  • The Hessian test: D > 0 with f_xx > 0 is a minimum, D > 0 with f_xx < 0 a maximum, D < 0 a saddle; eigenvalue signs generalize this to any dimension.
  • Lagrange multipliers solve grad f = lambda grad g with g = c; evaluate all candidates.
  • Gradient descent steps along -grad f; the learning rate must stay below 2 / L for a quadratic with largest curvature L.
  • sympy.solve, sympy.hessian and plain numpy loops cover these tools.

Next: Double and Triple Integrals.