Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 73 ยท Neural Networks From Scratch

Backpropagation by Hand

Work through backpropagation on a tiny network with real numbers: computational graph, chain rule, forward and backward values, checked with code.

Gradient descent needs one thing from the network: for every parameter, the slope of the loss with respect to that parameter. A real network has millions of parameters, buried under many layers, so you cannot just write the derivatives down. Backpropagation (backprop) is the procedure that computes all of them efficiently by applying the chain rule from the loss backward through the network.

Backprop has a reputation for being hard. It is not, once you do it with actual numbers on a network small enough to hold in your head. This lesson does exactly that: a two-parameter-per-layer network, every value computed by hand, and then verified with code. The next lesson, Backpropagation in NumPy, turns the same recipe into matrix code.

Backprop in four moves

Forward Compute and remember every intermediate value.
Loss Measure the error at the end.
Backward Pass a gradient back through each node.
Collect Each parameter now has its gradient.

The backward pass costs about as much as the forward pass.

1 / 2

The chain rule in one minute

Suppose y depends on u, and u depends on x. How fast does y change when you nudge x? The nudge is amplified by each link in the chain:

TEXT
dy/dx = (dy/du) * (du/dx)

If u changes 3 times as fast as x, and y changes 5 times as fast as u, then y changes 15 times as fast as x. A neural network is a long chain of such links, so the gradient for an early parameter is a product of local slopes along the way to the loss.

The practical trick of backprop is organization. Each node only needs to know two things: its own local derivative (how its output changes with its inputs) and the upstream gradient (how the loss changes with its output). It multiplies them and passes the result down to its inputs.

A tiny network

Our network has one input, one hidden unit with a ReLU, and one output with no activation:

TEXT
z1 = w1 * x + b1        (pre-activation of the hidden unit)
h  = relu(z1)           (hidden activation)
yhat = w2 * h + b2      (prediction)
L  = (yhat - y)^2       (squared-error loss for one example)

Pick concrete numbers: input x = 2, target y = 1, and parameters w1 = 0.5, b1 = 0.1, w2 = -0.4, b2 = 0.3.

flowchart LR
    x["x = 2"] --> M1(("ร—"))
    w1["w1 = 0.5"] --> M1
    M1 --> A1(("+"))
    b1["b1 = 0.1"] --> A1
    A1 -->|"z1"| R(("ReLU"))
    R -->|"h"| M2(("ร—"))
    w2["w2 = -0.4"] --> M2
    M2 --> A2(("+"))
    b2["b2 = 0.3"] --> A2
    A2 -->|"yhat"| S(("(yhat - y)ยฒ"))
    S --> L["L"]

The computational graph: each circle is one simple operation, and gradients flow through it from right to left.

Forward pass

Go left to right and write down every value, because the backward pass needs them.

  1. w1 * x = 0.5 * 2 = 1.0
  2. z1 = 1.0 + 0.1 = 1.1
  3. h = relu(1.1) = 1.1 (positive, so it passes through)
  4. w2 * h = -0.4 * 1.1 = -0.44
  5. yhat = -0.44 + 0.3 = -0.14
  6. L = (-0.14 - 1)^2 = (-1.14)^2 = 1.2996

Backward pass

Now go right to left, starting with dL/dL = 1.

The loss node. L = (yhat - y)^2, so the local derivative is 2 * (yhat - y) = 2 * (-1.14) = -2.28. Therefore dL/dyhat = -2.28. The negative sign says: raising the prediction would lower the loss, which makes sense because the prediction is below the target.

The output layer. yhat = w2 * h + b2. The addition node copies the upstream gradient to both inputs, so dL/db2 = -2.28. The multiplication node gives each input the upstream gradient times the other input:

  • dL/dw2 = dL/dyhat * h = -2.28 * 1.1 = -2.508
  • dL/dh = dL/dyhat * w2 = -2.28 * -0.4 = 0.912

The ReLU. Its local derivative is 1 where the input is positive and 0 elsewhere. Since z1 = 1.1 > 0, the gradient passes through unchanged: dL/dz1 = 0.912.

The first layer. Again an addition then a multiplication:

  • dL/db1 = 0.912
  • dL/dw1 = dL/dz1 * x = 0.912 * 2 = 1.824

(The gradient with respect to x itself would be 0.912 * 0.5 = 0.456, which we do not need because the input is data, not a parameter.)

That is all of backprop: one local rule per node, applied in reverse. Let the computer repeat the arithmetic and also check each answer with a brute-force test, nudging a parameter by a tiny amount and watching the loss.

Python The worked example, with a numerical check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two columns agree to the printed precision. That "nudge and measure" idea becomes a formal debugging tool in Gradient Checking.

One step of learning

With the gradients in hand, take one gradient-descent step with lr = 0.1 and see if the loss really falls.

Python One update lowers the loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prediction moves from -0.14 toward the target 1 and the loss shrinks at every step. All four parameters moved at once, each by an amount proportional to how much it mattered.

Why branches add

Our network was a straight line, but real graphs fork: one value feeds two later computations. For example, a = 3, p = a * 2, q = a ** 2, and r = p + q. How does r change with a? There are two routes from a to r, and both contribute:

TEXT
dr/da = (dr/dp)(dp/da) + (dr/dq)(dq/da) = 1 * 2 + 1 * (2a) = 2 + 6 = 8

The rule is: when a value is used in several places, add up the gradients arriving from each use. Forgetting this is the classic bug in hand-written backprop, and it appears whenever a layer is reused, as in residual connections and recurrent networks.

Python Two paths, two contributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a ReLU is "off"

Change the example slightly: set b1 = -2. Then z1 = 1.0 - 2 = -1.0, and relu(-1) = 0. The ReLU's local derivative is 0, so dL/dz1 = 0, and therefore both dL/dw1 and dL/db1 are 0. The first layer receives no learning signal at all for this example, because the unit is switched off. This is the seed of the "dying ReLU" problem from Activation Functions.

Also notice that h = 0 makes dL/dw2 = dL/dyhat * 0 = 0. Only b2 learns, shifting the output toward the target. Backprop respects the structure: a parameter only gets blame for an error to the extent that it influenced the prediction.

Summary of the recipe

  1. Run the forward pass and store every intermediate value.
  2. Start from the loss and compute dL/d(prediction).
  3. At each node, multiply the upstream gradient by the node's local derivative: addition passes it through, multiplication swaps in the other input, ReLU masks it.
  4. Where a value is reused, add the gradients.
  5. Read off the gradient for each parameter and update it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a multiplication node z = a * b send backward to a?

    dz/da = b, so by the chain rule the gradient for a is the upstream gradient times the other input, b.
  2. 2. In the worked example, why is dL/dyhat negative (-2.28)?

    The derivative of (yhat - y)^2 is 2 * (yhat - y) = 2 * (-1.14). A negative slope means moving the prediction up reduces the loss.
  3. 3. A value feeds two different later operations. How are its gradients combined?

    Each path from the value to the loss contributes separately, and the total derivative is the sum of those contributions.
  4. 4. If the pre-activation of a ReLU unit is negative, what gradient reaches the weights before it?

    The ReLU's local derivative is 0 for negative inputs, which multiplies everything upstream by 0.

Summary

  • Backprop is the chain rule applied from the loss backward: each node multiplies the upstream gradient by its local derivative.
  • Addition nodes copy the gradient, multiplication nodes swap in the other input, and a ReLU masks by whether its input was positive.
  • Always store forward values; the backward pass reuses them.
  • When a value is used more than once, its gradients add.
  • You can verify any gradient by nudging a parameter and measuring the change in loss.

Next lesson: Backpropagation in NumPy.