Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 111 Ā· Getting Started

Multivariable Calculus for Machine Learning

Partial derivatives, the gradient, the Hessian, Newton and quasi-Newton methods, and Lagrange multipliers, explained with short numerical examples for ML.

A machine learning model has many parameters, and training means finding the values that make a loss as small as possible. So the calculus you need is the calculus of functions with many inputs. You already met derivatives and the chain rule in linear algebra and calculus for ML. This lesson extends them to several variables and adds three tools that appear all over optimization: the Hessian (curvature), Newton and quasi-Newton methods (smarter steps) and Lagrange multipliers (optimization under constraints).

Partial derivatives and the gradient

For a function f(x, y), the partial derivative with respect to x, written āˆ‚f/āˆ‚x, is the ordinary derivative when you treat y as a constant. It answers: if I nudge only x, how fast does f change?

For example, take f(x, y) = x² + 3xy + 2y² - 4x. Then āˆ‚f/āˆ‚x = 2x + 3y - 4 and āˆ‚f/āˆ‚y = 3x + 4y.

Collecting all the partial derivatives in a vector gives the gradient, written āˆ‡f. Two facts make it the engine of learning:

  1. āˆ‡f points in the direction in which f increases fastest.
  2. Its length is the rate of increase in that direction.

So to reduce a loss you step opposite to the gradient. That is gradient descent: w ← w - Ī· āˆ‡L(w). At a minimum (or maximum, or saddle point) the gradient is the zero vector.

If a function maps many inputs to many outputs (a layer of a neural network, for instance), the matrix of all partial derivatives is the Jacobian. The chain rule for such functions is a product of Jacobians, which is exactly what backpropagation computes.

Curvature and the Hessian

The gradient says which way is down, but not how sharply the surface bends. The Hessian H is the matrix of second partial derivatives, H[i][j] = āˆ‚Ā²f / āˆ‚xi āˆ‚xj. It describes the local curvature:

  • If every eigenvalue of H is positive at a point where āˆ‡f = 0, the point is a minimum (the surface curves up in all directions).
  • If every eigenvalue is negative, it is a maximum.
  • If some are positive and some negative, it is a saddle point: up in one direction, down in another.

High-dimensional loss surfaces of neural networks have far more saddle points than local minima, which is one reason plain gradient descent can crawl through flat regions.

The ratio between the largest and smallest eigenvalue (the condition number) tells how stretched the valley is. A long, narrow valley makes plain gradient descent zigzag.

Python Gradient and Hessian with finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Hessian of this quadratic is constant, [[2, 3], [3, 4]], and its eigenvalues have mixed signs, so the stationary point of f is a saddle, not a minimum. Finite differences are fine for checking a formula; real libraries use automatic differentiation, which is exact and fast.

Newton's method: use the curvature

Gradient descent uses only the slope. Newton's method also uses the Hessian: it fits a parabola to the local surface and jumps to the bottom of that parabola.

TEXT
w ← w - H⁻¹ āˆ‡f(w)

On a quadratic loss this reaches the minimum in a single step, whatever the learning rate would have been. The next example compares 100 steps of gradient descent with one Newton step on a stretched valley (one direction 50 times steeper than the other).

Python One Newton step versus a hundred gradient steps

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The step size of gradient descent is limited by the steep direction (anything above 0.04 diverges), so progress along the flat direction is slow: after 100 steps the first coordinate is still 0.95 instead of 1. Newton rescales each direction by its curvature and lands on the minimum immediately.

The catch is cost. For a model with n parameters the Hessian has n² entries and inverting it takes about n³ operations. For a million parameters that is impossible, so deep learning uses first-order methods (see optimizers).

Quasi-Newton methods: curvature without the Hessian

Quasi-Newton methods such as BFGS and its low-memory sibling L-BFGS build an approximation of the (inverse) Hessian from the last few gradients, so they get most of Newton's speed without ever forming the matrix. They are the default for medium-sized smooth problems: scikit-learn's LogisticRegression uses lbfgs as its default solver, and SciPy's minimize uses BFGS.

The classic test is the Rosenbrock function, a curved, narrow valley whose minimum is at (1, 1).

Python Plain gradient descent against BFGS and conjugate gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

BFGS needs about 30 iterations to reach the minimum, and plain gradient descent is still crawling along the valley after 5,000 steps. Quasi-Newton methods need exact or well-behaved gradients and do not like noisy mini-batch gradients, which is why they are used for classical models and not for large neural networks.

Constraints and Lagrange multipliers

Sometimes you must optimize while obeying a rule: maximize an area given a fixed perimeter, find the best weights under a budget, find the widest margin that still classifies every point correctly. Lagrange multipliers turn a constrained problem into an unconstrained one.

To minimize f(x) subject to g(x) = 0, build the Lagrangian

TEXT
L(x, Ī») = f(x) - Ī» g(x)

and set all its partial derivatives to zero. The key insight is geometric: at the best point, the gradient of f must be parallel to the gradient of g (āˆ‡f = Ī» āˆ‡g), otherwise you could slide along the constraint and improve f. The multiplier Ī» tells how much the optimum would improve if you relaxed the constraint a little.

Worked example. Maximize x Ā· y subject to x + y = 10. Then āˆ‡f = (y, x) and āˆ‡g = (1, 1), so y = Ī» and x = Ī», hence x = y. With x + y = 10 this gives x = y = 5 and the maximum product 25.

Python Checking the Lagrange examples numerically

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Constraints show up throughout machine learning. The support vector machine is a constrained problem (maximize the margin while classifying correctly), and its Lagrangian leads to the "dual" form that makes the kernel trick possible. Ridge and lasso regression can be read as "minimize the error subject to a limit on the size of the weights", with the penalty strength playing the role of the multiplier. For inequality constraints the generalization is called the KKT conditions.

Check your understanding

Quick check Ā· 4 questions

of 4 answered You scored / 4 Ā· perfect!

  1. 1. What does the gradient of a loss function point toward?

    The gradient is the direction of steepest ascent. Gradient descent therefore moves in the opposite direction.
  2. 2. At a point where the gradient is zero, the Hessian has eigenvalues 3 and -1. What is the point?

    Mixed signs mean the surface curves up in one direction and down in another: a saddle.
  3. 3. Why is Newton's method rarely used to train large neural networks?

    Newton's step needs H inverse. Quasi-Newton methods approximate it, and deep learning prefers cheap first-order optimizers.
  4. 4. At the solution of a constrained problem, what is true about the gradients of f and of the constraint g?

    If they were not parallel, moving along the constraint would still improve f, so the point could not be optimal.

Summary

  • The gradient collects partial derivatives and points uphill; learning steps against it.
  • The Hessian holds second derivatives; its eigenvalues classify minima, maxima and saddle points.
  • Newton's method uses curvature to take better steps but is too costly in high dimensions; BFGS and L-BFGS approximate it.
  • Lagrange multipliers solve constrained optimization by making the gradient of the objective parallel to the gradient of the constraint.

Next: the first-order methods that scale to millions of parameters, in Optimizers: Momentum, RMSprop and Adam.