Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 76 ยท Math Foundations

Calculus and Optimization for Data Science

Understand derivatives, gradients and gradient descent with plain NumPy: minimize a loss by hand, choose a learning rate, fit a line step by step and avoid local minima.

Most machine-learning models learn by trial and improvement. They start with random settings, measure how wrong their predictions are, and then nudge the settings to be a little less wrong. Repeat that thousands of times and you have a trained model. Calculus is the tool that tells the model which way to nudge.

You do not need to solve integrals by hand. You need three ideas: a loss measures how bad the current settings are, a derivative tells you how the loss changes if you change one setting, and gradient descent follows those slopes downhill. This lesson builds each idea with small NumPy programs. It follows Linear Algebra for Data Science, where you saw X @ w as a prediction.

The loss: a number that says how wrong you are

Suppose a model predicts house prices and gets some of them wrong. We want a single number that summarises the errors. The most common choice is the mean squared error (MSE): square every error (so that positive and negative errors do not cancel and big errors count extra) and average them. A smaller loss means a better model.

Training means finding the weights that give the smallest loss. Think of the loss as a landscape: the weights are your position, the loss is your height, and you are looking for the lowest valley.

The derivative: the slope under your feet

The derivative of a function at a point is its slope there: how much the output changes when you move the input a tiny bit. For f(x) = xยฒ, the derivative is 2x. At x = 3 the slope is 6: the function is rising steeply; at x = 0 the slope is 0: you are at the bottom.

You can estimate any derivative numerically with the definition: (f(x + h) - f(x)) / h for a tiny h. This numerical derivative is slow, but it is a great way to check formulas.

Python Derivatives: formula against numerical estimate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The slope is negative to the left of -1.5 (the function is falling) and positive to the right (rising), so the lowest point is where it is exactly zero. This is the central rule of optimization: at a minimum, the slope is zero. A handful of derivative rules cover almost everything you will meet:

Function Derivative
c (a constant) 0
x^n n * x^(n-1)
a * f(x) a * f'(x)
f(x) + g(x) f'(x) + g'(x)
f(g(x)) (the chain rule) f'(g(x)) * g'(x)
e^x e^x
ln(x) 1 / x

The chain rule deserves attention, because a model is a chain of functions: weights feed into a prediction, which feeds into a loss. The chain rule says to multiply the slopes along the chain. Neural-network training (backpropagation) is exactly the chain rule applied efficiently.

The gradient: slopes in many directions

A model has many weights, not one. The gradient is the list of partial derivatives, one per weight: each entry says how much the loss changes if only that weight is nudged. The gradient points in the direction of the steepest increase, so the negative gradient points steepest downhill.

flowchart LR
    A["Weights w"] --> B["Predictions"]
    B --> C["Loss"]
    C --> D["Gradient:<br/>slope of loss for each weight"]
    D -->|"w = w - learning_rate * gradient"| A

Gradient descent: measure the loss, compute the slopes, take a small step downhill, repeat.

Gradient descent in one dimension

The rule is a single line:

w_new = w_old - learning_rate * slope

The learning rate is the step size. Let us minimize f(x) = (x - 4)^2 + 1, whose minimum is at x = 4:

Python Gradient descent on a parabola

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Steps start large, because the slope is steep, and shrink as the curve flattens near the bottom. That automatic slowing is why gradient descent settles instead of jumping around the minimum.

The learning rate matters

Too small and training crawls. Too large and each step overshoots, bouncing from one side of the valley to the other and getting worse instead of better. Try three values:

Python Too small, just right, too large

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With lr = 0.01 the point is still far from 4 after 20 steps. With 0.4 it arrives. With 1.05 every step overshoots by more than the previous distance and the value diverges (it explodes away from the minimum). In practice you try a few values, often on a log scale (0.1, 0.03, 0.01), and watch the loss curve: it should go down smoothly. The standard libraries also adapt the step size automatically (Adam, RMSprop), but the idea is the same.

Fitting a line by gradient descent

Now the real thing: learn the slope w and the intercept b of a line y = w*x + b. The loss is the MSE. Using the chain rule, the gradients are:

  • d(loss)/dw = mean(2 * (pred - y) * x)
  • d(loss)/db = mean(2 * (pred - y))
Python Linear regression trained by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After 300 steps the learned numbers are close to the exact least-squares solution; the intercept is still creeping toward it, because gradient descent stops at "good enough" rather than at the exact answer. For plain linear regression there is a direct formula, so gradient descent is not needed. But for logistic regression, neural networks and most other models there is no direct formula, and gradient descent (usually with small batches of data at a time, called stochastic gradient descent) is how every one of them learns.

Local minima, scaling and other practical points

The parabola has one valley. Real loss landscapes can have several, plus flat plateaus and saddle points. Gradient descent only goes downhill, so it can settle in a local minimum that is not the lowest point overall:

Python Two valleys: the start decides the result

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Starting on the left finds the deeper valley (loss about -3.5); starting on the right gets trapped in the shallower one (about -1.1). Convex problems such as linear and logistic regression have a single valley, so this cannot happen. For non-convex models, people run training several times from different random starts, use momentum-based optimizers, and accept "good enough" valleys.

Practical rules:

  • Scale your features. If one feature is in the thousands and another is between 0 and 1, the landscape is a long narrow canyon and gradient descent zig-zags slowly. Standardizing features makes the bowl round.
  • Watch the loss curve. It should decrease. If it rises or becomes nan, the learning rate is too high.
  • Use a validation set to decide when to stop; a lower training loss does not always mean a better model (see Train, Test and Validation).
  • Numerical derivatives (finite differences) are for checking your formulas. Libraries such as PyTorch and JAX compute exact gradients automatically.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is true about the slope of a smooth function at its minimum?

    At the bottom of a smooth valley the function is neither rising nor falling, so the slope is zero.
  2. 2. In gradient descent, why do we subtract the gradient?

    The gradient points uphill. Moving against it lowers the loss.
  3. 3. A training run's loss grows every step and then becomes nan. What is the most likely cause?

    Steps larger than the distance to the minimum overshoot further each time, so the values explode.
  4. 4. What does the chain rule let you do?

    A model is a chain of functions from weights to loss. The chain rule multiplies their slopes, which is what backpropagation does.
  5. 5. Why does feature scaling help gradient descent?

    Unscaled features create a long narrow canyon in which descent zig-zags. Scaling makes the valley more circular.

Summary

  • A loss (such as MSE) measures how wrong a model is; training finds weights that minimize it.
  • A derivative is a slope; at a minimum the slope is zero. The chain rule multiplies slopes along a chain of functions.
  • The gradient lists the slope for every weight; the negative gradient points downhill.
  • Gradient descent repeats w = w - learning_rate * gradient. Too small a learning rate crawls, too large diverges.
  • Linear regression can be trained by hand with two gradients and matches the exact least-squares answer.
  • Non-convex losses have local minima; scale features, watch the loss curve and validate on held-out data.

Next: Collecting Data from APIs and Web Pages moves from maths to getting data.