Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 73 · First Steps

The Math You Need

A practical refresher of the math behind neural networks: vectors, matrices, dot products, derivatives, gradients and the chain rule, checked with NumPy.

Deep learning rests on a surprisingly small amount of mathematics. You need to multiply matrices, understand what a slope is, and apply one rule from calculus, the chain rule, over and over. That is nearly all of it. This lesson covers just those pieces, each with a short NumPy check, so that later lessons can use them without stopping to explain.

If you want the full story on arrays and matrices, the NumPy tutorial has it: see Vectors and the Dot Product and Matrix Operations. Here we concentrate on how these tools show up inside neural networks.

Vectors and dot products

A vector is an ordered list of numbers. In deep learning a vector might be the pixel values of an image, the measurements describing a house, or the weights of one neuron. The dot product of two vectors of equal length multiplies them element by element and adds the results:

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ

The dot product measures how much two vectors point the same way. It is large and positive when they agree, zero when they are perpendicular, and negative when they oppose each other. A neuron is, at its heart, a dot product between its inputs and its weights.

Python Dot products measure alignment

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Matrices as layers

A matrix is a grid of numbers. Multiplying a vector by a matrix produces a new vector, and each output number is a dot product of the input with one column of the matrix. That is exactly what a layer of neurons does: many dot products at once.

If x has 3 numbers and the matrix W has shape (3, 4), then x @ W has 4 numbers. The rule to remember is that inner dimensions must match and the outer dimensions give the result: (3,) @ (3, 4) gives (4,).

Stack many inputs as the rows of a matrix X with shape (n_samples, 3) and one multiplication processes the whole batch: (n, 3) @ (3, 4) gives (n, 4).

Shape rules to memorize

Inner match (n, a) @ (a, b) works; the a's must agree.
Outer survive The result has shape (n, b).
Transpose flips A.T swaps rows and columns, (a, b) becomes (b, a).

When a network crashes with a shape error, check these three rules first.

1 / 2
Python A layer is a matrix multiplication

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Derivatives: how fast is it changing?

The derivative of a function at a point is its slope there: how much the output changes per tiny change in the input. For f(x) = x² the derivative is 2x. At x = 3 the slope is 6, so moving x up by 0.001 raises f by about 0.006.

You do not need to memorize many derivatives. These cover nearly everything in this tutorial:

Function Derivative
c (a constant) 0
a·x a
x² 2x
eˣ eˣ
ln x 1/x

A derivative also tells you which way is downhill. If the slope is positive, the function rises as x increases, so decreasing x lowers it. That is the whole idea behind gradient descent.

You can always check a derivative numerically by nudging the input a tiny bit and measuring the rise over the run. This finite difference trick returns later in Gradient Checking.

Python Checking a derivative with a nudge

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Partial derivatives and gradients

Neural networks have many inputs and parameters, so a function usually depends on several variables. A partial derivative treats all variables but one as constants and asks how the output changes with that one. For f(x, y) = x² + 3y², the partial derivative with respect to x is 2x and with respect to y is 6y.

Collect all the partial derivatives into a vector and you get the gradient, written ∇f. The gradient is an arrow that points in the direction of steepest increase, and its length says how steep that is. To lower the function, step the opposite way: along −∇f.

In training, f is the loss and the variables are the parameters, so the gradient tells you how to change every weight to reduce the error.

Python Gradient arrows point uphill

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice in the picture that the arrows are longer where the surface is steeper, and every arrow crosses the contour lines at a right angle.

The chain rule

A neural network is a chain of functions: the output of one layer feeds the next. The chain rule says how to differentiate such a chain. If y = f(g(x)), then

dy/dx = f′(g(x)) · g′(x)

In words: the rate at which the end result changes with x equals the rate at which f changes with its input, times the rate at which g changes with x. The rates multiply along the chain.

Take y = (3x + 1)². Let u = 3x + 1, so y = u². Then dy/du = 2u and du/dx = 3, giving dy/dx = 2u · 3 = 6(3x + 1).

flowchart LR
    X["x"] -->|"du/dx = 3"| U["u = 3x + 1"]
    U -->|"dy/du = 2u"| Y["y = u²"]

Multiply the local slopes along the path from x to y.

Python The chain rule, verified

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Backpropagation is the chain rule applied to a network with millions of links, organized so that no work is repeated. When you reach Backpropagation by Hand, nothing new will appear except bookkeeping.

Two more tools

The sum rule and averages. The derivative of a sum is the sum of the derivatives. Because a loss is usually the average error over many examples, its gradient is the average of the per-example gradients.

Shapes of gradients. The gradient of a loss with respect to a parameter always has the same shape as that parameter: a (3, 4) weight matrix gets a (3, 4) gradient. When your gradient code produces any other shape, it is wrong. This single rule catches a large share of bugs.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What shape does (6, 3) @ (3, 5) produce?

    The inner dimensions (3 and 3) match and disappear; the outer dimensions 6 and 5 form the result.
  2. 2. The gradient of a function at a point is...

    The gradient points uphill. Gradient descent steps along the negative gradient to go downhill.
  3. 3. If y = f(g(x)), the derivative dy/dx equals...

    The chain rule multiplies the derivative of the outer function, evaluated at the inner function's value, by the derivative of the inner function.
  4. 4. A weight matrix has shape (4, 7). What shape must its gradient have?

    A gradient always has exactly the shape of the parameter it belongs to, one slope per entry.

Summary

  • A dot product measures alignment; a layer is many dot products done as one matrix multiplication.
  • Shape rule: inner dimensions match, outer dimensions form the result.
  • A derivative is a slope; a gradient is the vector of partial derivatives and points uphill.
  • Gradient descent moves against the gradient to lower the loss.
  • The chain rule multiplies local slopes along a chain of functions; backpropagation is this rule applied systematically.
  • A gradient has the same shape as its parameter.

Next: The Artificial Neuron.