Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 40 ยท Tensors and Automatic Differentiation

Gradient Descent from Scratch

Build gradient descent step by step: learning rate effects, mini-batches, a linear regression training loop in NumPy and the same custom loop in TensorFlow.

You now have the pieces: tensors, variables and gradients. This lesson puts them together into the training loop that sits at the heart of every Keras fit call. We will fit a linear model with gradient descent in NumPy, explore what the learning rate and batch size do, and then read the identical loop written with tf.GradientTape. After this lesson, model.fit will no longer be a black box.

The idea

Imagine standing on a hilly landscape in fog, wanting to reach the lowest point. You cannot see far, but you can feel the slope under your feet. Take a small step downhill, feel again, repeat. The height is the loss, your position is the weights and the slope is the gradient:

weights = weights - learning_rate * gradient

The learning rate sets the step size. Too small and you crawl; too large and you leap over the valley and may fly off the map.

flowchart TB
    A["Initialize weights"] --> B["Forward: predictions"]
    B --> C["Loss"]
    C --> D["Gradient of loss<br/>w.r.t. weights"]
    D --> E["weights -= lr * gradient"]
    E --> F{"Converged or<br/>out of epochs?"}
    F -->|no| B
    F -->|yes| G["Trained model"]

Gradient descent is a loop, and each lap is one update.

Linear regression by hand

Model: y_hat = X @ w + b. Loss: mean squared error, mean((y_hat - y)**2). The gradients come from calculus, or from autodiff:

  • dL/dw = (2 / n) * X.T @ (y_hat - y)
  • dL/db = (2 / n) * sum(y_hat - y)

We generate data from a known rule, y = 3*x1 - 2*x2 + 5 plus a little noise, so we can check that training recovers the numbers.

Python Stand-in: full-batch gradient descent (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls quickly and the weights approach [3, -2] and 5. The small leftover error comes from the noise we added.

The learning rate matters most

Run the same loop with different learning rates and watch the final loss. This is the single most useful experiment in deep learning.

Python Stand-in: three learning rates compared (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see: a tiny rate barely moves in 50 epochs; 0.1 converges nicely; 0.9 converges but overshoots back and forth, ending with a worse loss; 1.1 diverges (the loss explodes). For this problem the safe limit depends on the data scale, which is one more reason to normalize your inputs.

A reliable approach for Keras: start with Adam at learning_rate=1e-3, and tune by factors of 3 or 10 (3e-4, 1e-2). Watch the loss curve in the first few epochs.

Mini-batch stochastic gradient descent

Computing the gradient over all examples each step is exact but slow for big data. Stochastic gradient descent (SGD) uses one example; mini-batch SGD uses a small batch (32 to 256). The gradient is a noisy estimate of the true one, but you can take many more steps per second, and the noise often helps escape poor spots.

Method Examples per update Updates per epoch Behavior
Full batch all n 1 smooth, slow, memory heavy
Mini-batch 32-256 n / batch the practical default
Stochastic 1 n very noisy
Python Stand-in: mini-batch SGD (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that after a single epoch the model is already close, because each epoch contains 32 updates here, not one.

The same loop with TensorFlow

With tf.GradientTape you do not derive any formulas. Read this and compare with the NumPy version: only the gradient computation changed.

Python
import tensorflow as tf

w = tf.Variable(tf.zeros((2, 1)))
b = tf.Variable(0.0)
lr = 0.05

dataset = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(1000).batch(32)

for epoch in range(6):
    for xb, yb in dataset:
        with tf.GradientTape() as tape:
            y_hat = xb @ w + b
            loss = tf.reduce_mean((y_hat[:, 0] - yb) ** 2)
        grad_w, grad_b = tape.gradient(loss, [w, b])
        w.assign_sub(lr * grad_w)
        b.assign_sub(lr * grad_b)
    print(epoch, float(loss))

The dataset yields shuffled batches; the tape records the forward pass; tape.gradient returns gradients; assign_sub applies the update.

The same loop with Keras optimizers

You can let a Keras optimizer do the update, which is what fit does internally:

Python
optimizer = keras.optimizers.SGD(learning_rate=0.05)

with tf.GradientTape() as tape:
    loss = compute_loss(xb, yb)
grads = tape.gradient(loss, [w, b])
optimizer.apply_gradients(zip(grads, [w, b]))

Swapping SGD for Adam changes only that one line, which is why optimizers are interchangeable. Optimizers explains the differences.

Pitfalls

  • Divergence: the loss becomes nan or inf. Lower the learning rate, scale your inputs, clip gradients.
  • Plateaus: tiny gradients make progress slow. Try a larger rate, a better optimizer or a different initialization.
  • Local minima and saddle points: in high dimensions saddle points are more common than bad local minima, and noisy mini-batches help escape them.
  • Not shuffling: batches with similar examples give biased gradients.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the gradient descent update rule?

    We move against the gradient, which is the direction of steepest increase of the loss.
  2. 2. What happens when the learning rate is much too large?

    Oversized steps overshoot the minimum, so the loss can grow, ending in inf or nan.
  3. 3. Why is mini-batch SGD the practical default?

    Small batches trade exactness for speed and memory, and work very well in practice.
  4. 4. In the TensorFlow loop, which line applies the update?

    assign_sub writes the new weight value back into the variable.

Summary

  • Gradient descent repeats: forward, loss, gradient, weights -= lr * gradient.
  • The learning rate is the most important setting; too small crawls, too large diverges.
  • Mini-batch SGD gives many cheap noisy updates per epoch.
  • A custom TensorFlow loop is a GradientTape plus assign_sub or optimizer.apply_gradients.
  • Keras fit wraps this same loop.

Next: Building a Dense Layer from Scratch