Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 39 · The Gradient and Directional Derivatives

Gradient Descent and Steepest Ascent in Practice

Use the negative gradient to find minima numerically: implement gradient descent, choose step sizes, see zig-zags and divergence, and compare with scipy.

Because -∇f points in the direction of steepest decrease, we can find a minimum of a function by repeatedly stepping downhill. This simple idea, gradient descent (or steepest descent), is used everywhere from fitting curves to training neural networks. In this lesson you implement it from scratch and watch what goes right and wrong.

The algorithm

Start at a point p₀. Repeat:

Text
p_{k+1} = p_k - α ∇f(p_k)

The number α > 0 is the step size (in machine learning, the learning rate). Stop when |∇f| is tiny (we are near a critical point) or after a maximum number of steps.

Why it works: for a small step, f(p - α∇f) ≈ f(p) - α|∇f|², which is smaller than f(p) whenever ∇f ≠ 0. Why the step size matters: too large and you overshoot the valley.

Worked example: f(x, y) = x² + 3y². The gradient is ⟨2x, 6y⟩. Start at (4, 2) with α = 0.1:

Text
p₁ = (4, 2) - 0.1·(8, 12) = (3.2, 0.8)
p₂ = (3.2, 0.8) - 0.1·(6.4, 4.8) = (2.56, 0.32)

Each coordinate is multiplied by a constant factor: x by 0.8, y by 0.4. Both shrink toward 0, the minimum.

Python Gradient descent from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step size matters

For f = x² + 3y² the update multiplies y by 1 - 6α each step. This has absolute value less than 1 only if 0 < α < 1/3. If α = 0.4, then 1 - 2.4 = -1.4, and y grows in size at each step: the algorithm diverges. If α is tiny, convergence is safe but very slow.

Python Compare three step sizes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With α = 0.34 the y-factor is 1 - 6(0.34) = -1.04, so f slowly grows: the iterates oscillate across the valley and drift away.

Zig-zagging in narrow valleys

When the level curves are long thin ellipses (say f = x² + 25y²), the gradient points mostly across the valley rather than along it, so the path zig-zags. The ratio of the steepest to the gentlest curvature (here 50/2 = 25) controls how slow descent is. Rescaling the variables or using momentum or second-order methods helps.

Python Path of descent on a narrow valley

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Steepest ascent and comparing with a library

To find a maximum, step along +∇f (gradient ascent), or equivalently minimize -f. Here we maximize f(x, y) = -(x - 1)² - (y + 2)² + 5, whose peak is at (1, -2) with value 5, and check against scipy.optimize.minimize applied to -f.

Python Gradient ascent vs scipy

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gradient descent

Start anywhere
Step along -∇f
Repeat Until the gradient is tiny
1 / 2

Gradient descent finds a local minimum (or sometimes a saddle point), not necessarily the global one. For functions with several valleys the start point decides which one you reach.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. The gradient descent update rule is

    We move against the gradient to decrease f.
  2. 2. For f = x² + 3y², gradient descent with step α converges only if

    The y-coordinate is multiplied by 1 - 6α each step, and |1 - 6α| < 1 requires 0 < α < 1/3.
  3. 3. Zig-zagging typically occurs when

    In a narrow valley the gradient points mostly across the valley.
  4. 4. To find a maximum with the same idea you

    Gradient ascent moves in the direction of steepest increase.

Summary

  • Gradient descent repeats p ← p - α ∇f(p); gradient ascent uses +.
  • The step size must be small enough: too large diverges, too small crawls.
  • Narrow valleys cause zig-zags; the result is a local, not necessarily global, minimum.
  • Results should be checked against a trusted optimizer such as scipy.optimize.minimize.

Next: Critical Points and the Second Derivative Test