Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 68 ยท Multivariable Calculus

Gradient and Directional Derivatives

Understand the gradient vector, directional derivatives, steepest ascent, level curves and tangent planes, with runnable sympy, numpy and contour plot examples.

A partial derivative tells you the slope of a surface if you walk exactly along the x axis or exactly along the y axis. But a hiker on a hillside can walk in any direction, and the slope depends on which way they face. The natural question is: in which direction does the ground rise fastest, and how steep is it there?

The answer is one of the most useful objects in all of applied mathematics, the gradient. It packs the partial derivatives into a single vector and turns out to point straight uphill. The same vector drives gradient descent in machine learning, heat flow in physics and the shape of electric fields. This lesson builds on Partial Derivatives and Vectors in 2D and 3D, because the key formula is simply a dot product.

The gradient vector

For f(x, y), the gradient is the vector of partial derivatives:

grad f = (f_x, f_y).

For three variables it has three components. It is often written โˆ‡f (read "del f" or "nabla f"). At each point of the plane, grad f gives one arrow, so the gradient is a vector field.

For f(x, y) = x**2 + 2*y**2, we get grad f = (2x, 4y). At the point (1, 1) the gradient is (2, 4): the function grows twice as fast in the y direction as in the x direction.

Python Compute a gradient symbolically and numerically

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numeric_gradient helper works for any function of a vector, which makes it a reliable way to check a gradient you derived by hand.

Directional derivatives

Pick a direction by choosing a unit vector u = (u1, u2). The directional derivative of f at a point in direction u is the rate of change of f as you move along u at unit speed:

D_u f = limit as h goes to 0 of ( f(p + h*u) - f(p) ) / h.

You never need to evaluate that limit directly. When f is differentiable, it equals a dot product:

D_u f = grad f . u.

The partial derivatives are the special cases u = (1, 0) and u = (0, 1). The vector u must have length 1; if you are given a direction like (3, 4), normalize it to (3/5, 4/5) first. Forgetting to normalize is the most common mistake with this formula.

Python Directional derivative: formula versus finite difference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both numbers agree at 4.4 (that is 2*0.6 + 4*0.8), which confirms the formula.

Why the gradient points uphill

Since D_u f = grad f . u = |grad f| |u| cos(theta) = |grad f| cos(theta), where theta is the angle between u and the gradient, the directional derivative depends only on that angle. Read off three cases.

  • theta = 0 (walk along the gradient): cos = 1, so the rate is |grad f|. This is the maximum: the gradient points in the direction of steepest ascent, and its length is the steepest slope.
  • theta = 180 degrees (walk against it): the rate is -|grad f|, the steepest descent. This is the reason gradient descent subtracts the gradient.
  • theta = 90 degrees (walk perpendicular): the rate is 0. You walk along a contour line, staying at the same height.
flowchart TD
    G["grad f at a point"] --> A["Along grad f<br/>fastest increase: |grad f|"]
    G --> B["Against grad f<br/>fastest decrease: -|grad f|"]
    G --> C["Perpendicular to grad f<br/>no change: along a level curve"]

The angle between your direction and the gradient decides the rate of change: |grad f| cos(theta).

Try it by brute force: sample many directions on a circle and see which one gives the largest slope.

Python Find the steepest direction by sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curve is a cosine wave whose peak height equals |grad f| = sqrt(20), about 4.4721, and whose peak sits at the gradient's own angle, about 63.4 degrees. The sampled peak prints 63.0 only because the angles were sampled in steps of one degree.

Level curves and the gradient

A level curve of f is the set of points where f(x, y) = c for a constant c. It is the same thing as a contour line on a map. Because f does not change along a level curve, the directional derivative along its tangent is zero, which means the tangent is perpendicular to the gradient. So:

The gradient at a point is always perpendicular to the level curve through that point.

Cartographers know this intuitively: the steepest path crosses the contour lines at right angles. The closer the contour lines are packed, the larger |grad f|.

Python Gradient arrows cross contour lines at right angles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All arrows point outward and away from the centre, where the function is smallest, and they are longer where the contours are tighter. The level curves here are ellipses.

Tangent planes and linear approximation

For a function of one variable, the tangent line at a is y = f(a) + f'(a)(x - a). For z = f(x, y) the matching object is a tangent plane at the point (a, b):

z = f(a, b) + f_x(a, b) (x - a) + f_y(a, b) (y - b).

Equivalently, z = f(p0) + grad f(p0) . (p - p0). The plane touches the surface at p0 and tilts exactly as the surface does there. Near that point it is a very good approximation, which is the idea behind Linearization and Newton's Method, extended to more variables.

For f = x**2 + 2*y**2 at (1, 1): f(1, 1) = 3, f_x = 2, f_y = 4, so the plane is z = 3 + 2(x - 1) + 4(y - 1).

Python Tangent plane as a linear approximation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error shrinks much faster than the step: the error scales with the square of the step (doubling the step from 0.5 to 1.0 makes the error four times bigger, 0.75 to 3.0). That is why tangent planes work well for small changes and badly for large ones.

For a surface defined implicitly by F(x, y, z) = 0, the gradient grad F is a normal vector to the surface. This is the quickest way to write down the tangent plane to a sphere or an ellipsoid.

Gradient in three variables

Nothing changes in 3D except for one extra component. For a temperature field T(x, y, z), grad T points toward the hottest direction, and heat flows opposite to it. sympy handles it the same way: differentiate with respect to each variable and collect the results.

Python A gradient in three variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At (1, 1, 1) the gradient is (-2, -4, -6), which points back toward the hot origin: that is the direction of fastest temperature rise. Heat flows the opposite way, along the normalized (2, 4, 6) printed on the last line.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the gradient of f(x, y) = x^2 + 2y^2?

    The gradient collects the partial derivatives: f_x = 2x and f_y = 4y. It is a vector, not a single number.
  2. 2. In which direction does f increase fastest at a point?

    D_u f = |grad f| cos(theta) is largest when theta = 0, so the steepest ascent is along the gradient.
  3. 3. The gradient at (2, 1) is (4, 4). What is the directional derivative in the direction of the vector (3, 4)?

    First normalize (3, 4) to (0.6, 0.8). Then D_u f = 40.6 + 40.8 = 5.6. Using (3, 4) without normalizing would give 28.
  4. 4. How is the gradient related to the level curve through the same point?

    f does not change along a level curve, so the directional derivative along its tangent is zero, which means the gradient is perpendicular to the tangent.
  5. 5. What is the tangent plane to z = f(x, y) at (a, b)?

    The plane passes through the point on the surface and has slopes f_x and f_y in the two axis directions.

Summary

  • The gradient grad f = (f_x, f_y, ...) collects all partial derivatives into one vector.
  • The directional derivative is D_u f = grad f . u, where u must be a unit vector.
  • The gradient points in the direction of steepest ascent, and its length |grad f| is the steepest slope.
  • It is perpendicular to level curves; tightly packed contours mean a large gradient.
  • The tangent plane z = f(p0) + grad f(p0) . (p - p0) approximates the surface near p0.
  • Moving against the gradient is steepest descent, the engine of optimization.

Next: Optimization in Several Variables.