Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 38 · Getting Started

Gradients and Hessians

Review the calculus you need: partial derivatives, the gradient, the Hessian matrix and the chain rule. Check your formulas with numerical differences in Python.

Optimization algorithms navigate using slopes. In one variable the slope is the derivative. With many variables you need two objects: the gradient (a vector of first derivatives) and the Hessian (a matrix of second derivatives). This lesson reviews both, shows how to verify your formulas numerically, and previews how they tell you where a minimum is.

The gradient

For f(x₁, x₂, ..., xₙ), the partial derivative ∂f/∂xᵢ is the slope when only xᵢ changes and the others are held fixed. The gradient stacks them:

Text
∇f(x) = ( ∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ )

Two facts to remember:

  1. ∇f(x) points in the direction of steepest increase of f.
  2. Its length tells you how steep that rise is. So -∇f(x) is the direction of steepest decrease: the heart of gradient descent.

Example: f(x, y) = x² + 3xy + y². Then ∂f/∂x = 2x + 3y and ∂f/∂y = 3x + 2y, so ∇f = (2x + 3y, 3x + 2y).

Checking a gradient numerically

Wrong gradients are the most common bug in optimization code. A finite-difference check compares your formula with (f(x + h·eᵢ) - f(x - h·eᵢ)) / (2h) for a tiny h, where eᵢ is the unit vector along axis i.

Python Gradient check with central differences

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both lines should agree to about six digits. If they do not, the formula has a mistake.

The Hessian

The Hessian ∇²f(x) is the matrix of all second partial derivatives, with entry (i, j) equal to ∂²f / (∂xᵢ ∂xⱼ). For smooth functions it is symmetric. For our example:

Text
∇²f = [[2, 3],
       [3, 2]]

The Hessian describes curvature. In one variable, f''(x) > 0 means the curve bends upward like a bowl. In many variables, the equivalent condition is that the Hessian is positive definite (all eigenvalues positive), which means the surface bends upward in every direction.

Python Eigenvalues reveal the shape

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first matrix has one positive and one negative eigenvalue (-1 and 5): the surface curves up along one direction and down along another, a saddle. The bowl has two positive eigenvalues.

Conditions for a minimum

For a smooth function with no constraints:

  • First-order necessary condition: if x* is a local minimum, then ∇f(x*) = 0.
  • Second-order sufficient condition: if ∇f(x*) = 0 and ∇²f(x*) is positive definite, then x* is a strict local minimum.

Let us apply this. Take f(x, y) = x² + xy + y² - 3x. Setting ∇f = (2x + y - 3, x + 2y) = 0 is a linear system.

Python Solve for the stationary point

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The point is (2, -1). The Hessian [[2,1],[1,2]] has eigenvalues 1 and 3, both positive, so it is a strict local minimum (and in fact the global one, since this function is convex).

The chain rule for vectors

Machine-learning objectives are compositions. If f(x) = φ(aᵀx), with a a fixed vector and φ a one-variable function, then ∇f(x) = φ'(aᵀx) · a. For f(x) = ½‖Ax - b‖² (least squares), the gradient is Aᵀ(Ax - b) and the Hessian is AᵀA. We will use these two formulas many times.

You do not need symbolic algebra for every problem. Automatic differentiation libraries compute gradients for you, but the finite-difference check remains useful for testing them.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. The gradient of f at x points in the direction of

    ∇f is the direction of fastest increase; we use -∇f to go downhill.
  2. 2. For f(x, y) = x² + 3xy + y², what is ∂f/∂y?

    Differentiate with respect to y holding x fixed: 3x + 2y.
  3. 3. If ∇f(x*) = 0 and the Hessian is positive definite, x* is

    Positive definite curvature in every direction makes the point a strict local minimum.
  4. 4. The Hessian of f(x) = ½‖Ax - b‖² is

    The gradient is Aᵀ(Ax - b); differentiating again gives AᵀA.

Summary

  • The gradient is the vector of partial derivatives; it points uphill.
  • The Hessian holds second derivatives and describes curvature; positive eigenvalues mean bowl-shaped.
  • A minimum needs a zero gradient; a positive definite Hessian makes it a strict local minimum.
  • Always check gradients against finite differences.

Next: Your First Solvers: Grid Search and Newton