Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 111 · Getting Started

Math for Machine Learning - Linear Algebra and Calculus

Learn the math behind machine learning in NumPy: vectors, dot products, matrices, least squares, eigenvectors, derivatives, gradients and the chain rule.

Machine learning looks like magic until you see how little math sits underneath it. Almost every model in this course is built from two ideas. Linear algebra describes data and predictions: a row of features is a vector, a table is a matrix, and a prediction is usually a dot product. Calculus describes learning: a loss measures how wrong the model is, and the derivative tells you which way to nudge each number to make it less wrong.

You do not need a math degree. This lesson keeps the formulas short and checks every one of them in NumPy, so you can see that the symbols and the code are the same thing. If a section feels abstract, run the example and change the numbers. Everything here reappears later, in Linear Regression, Gradient Descent and PCA.

Two kinds of math

Linear algebra Vectors and matrices hold data and weights.
Calculus Derivatives say how to improve a model.
NumPy One line of code per formula.
1 / 2

Vectors and the dot product

A vector is an ordered list of numbers. In machine learning a vector is usually one example: [3, 10, 2] could be the bedrooms, age and distance to the city of one house. In NumPy it is a one-dimensional array.

Two operations matter most. The norm is the length of a vector: for [3, 4] it is sqrt(3² + 4²) = 5. The dot product multiplies two vectors position by position and adds the results: [3, 4] · [4, 3] = 3·4 + 4·3 = 24. In NumPy you write a @ b.

The dot product measures how much two vectors point the same way. Divide it by both lengths and you get the cosine similarity, a number between -1 (opposite) and 1 (same direction); 0 means the vectors are at right angles. Text search and recommenders compare vectors this way. The dot product is also the heart of a linear model: a prediction is weights · features + bias.

Python Vectors and dot products

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Vector c is perpendicular to a, so their dot product is 0, and -a points exactly the other way, so its cosine is -1.

Matrices: many examples at once

A matrix is a table of numbers. Your feature table X has one row per example and one column per feature, so its shape is (n_samples, n_features). This is the same shape scikit-learn expects in fit(X, y).

Multiplying a matrix by a vector, X @ w, takes the dot product of every row with w in one go. That is how a linear model predicts for a thousand houses with a single line. Three more tools come up constantly:

  • Transpose (X.T) flips rows and columns.
  • Inverse (np.linalg.inv(A)) is the matrix that undoes A, so A @ inv(A) is the identity matrix. Not every matrix has one.
  • Solve (np.linalg.solve(A, b)) finds x in A @ x = b. It is faster and more accurate than computing the inverse yourself.

These combine into the oldest trick in machine learning: least squares. To fit a line to data you want the weights w that minimize the squared error ‖Xw − y‖². Setting the derivative to zero gives the normal equation, XᵀX w = Xᵀy, which is just a linear system you can hand to solve. A column of ones in X lets the model learn an intercept.

Python Matrices, solving and least squares

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two least-squares results agree and land near the true values 5, 2 and -3 (intercept first); the small gap is the noise we added. In practice LinearRegression calls a routine like lstsq, because it stays accurate when features are almost redundant.

Shapes are the number one source of bugs. When A @ B fails, print A.shape and B.shape: the last dimension of A must equal the first dimension of B.

Eigenvectors and the covariance matrix

Most matrices stretch and rotate vectors. An eigenvector of a matrix is a special direction that is only stretched, never turned, and the eigenvalue is the stretch factor: A v = λ v.

Why does a machine learning course care? The covariance matrix of your features describes the shape of the data cloud, and its eigenvectors are the axes of that cloud, with eigenvalues giving the variance along each axis. PCA keeps the axes with the largest eigenvalues. The singular value decomposition (SVD) reaches the same axes directly from the centered data, and is what libraries actually use.

Python Eigenvectors and the covariance matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

We built the data by tilting an ellipse by 30 degrees, and the first eigenvector recovers that tilt. The eigenvalues (9.44 and 0.49) are close to the variances we built in (3² = 9 and 0.7² ≈ 0.5), and the SVD route gives the same numbers.

Derivatives: how fast does it change?

The derivative of a function is its slope: how much the output changes when the input changes a tiny bit. For f(x) = x³ − 2x the derivative is 3x² − 2. If you forget a formula, you can always check it with a numerical derivative: evaluate the function a tiny step on both sides and divide, (f(x+h) − f(x−h)) / 2h. Professionals do this too, under the name gradient checking.

A loss function usually has many inputs, one per weight. The vector of slopes, one per weight, is the gradient. It points in the direction where the loss grows fastest, so to learn you step the opposite way:

w ← w − learning_rate × gradient

That one line is gradient descent, explained in depth in Gradient Descent.

The chain rule

Models are built by feeding the output of one step into the next, and the chain rule tells you how to differentiate such a chain: multiply the slopes of the links. For the loss (w·x − y)² on a single example, the inner step is w·x − y (slope x with respect to w) and the outer step is squaring (slope 2·(w·x − y)). Multiply them: d loss / d w = 2·(w·x − y)·x.

For linear regression with many examples, the mean squared error is mean((Xw − y)²) and the same logic gives a gradient you can write in matrix form: (2/n) · Xᵀ(Xw − y). The prediction error Xw − y is multiplied by the features, which says something intuitive: a weight changes most when its feature is large and the error is large. Neural networks apply this same chain rule through many layers, and backpropagation is simply the chain rule organized so that no work is repeated.

The example below checks each formula against a numerical derivative and then runs gradient descent from zero weights. The loss should fall steadily and the weights should approach the true values 5, 2 and -3.

Python Derivatives, the chain rule and gradient descent

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two lines show formula and numeric slope agree to four decimals. Gradient descent then turns a loss of about 37 into about 0.21 within 50 steps, which is the level of the noise we added (variance 0.25), without ever solving the normal equation. For linear regression that is a detour, but the same loop trains models that have no closed-form answer.

Before trusting a hand-written gradient, compare it with the numerical one on a few random points. A wrong gradient is silent: training runs, the loss even drops a little, and the model is just worse than it should be.

Orthogonal projections

Two vectors are orthogonal when their dot product is zero: they point in perpendicular directions and share no "information". The orthogonal projection of a vector b onto a direction a is the part of b that lies along a; what is left over is perpendicular to a:

TEXT
projection of b onto a  =  (a · b) / (a · a) × a        (a number times a)
residual                =  b - projection               (always orthogonal to a)

Projections are everywhere in machine learning:

  • Linear regression is a projection. The predictions ŷ = Xw are the closest point to y in the space spanned by the columns of X, and the residual vector y - ŷ is orthogonal to every column of X. That is why the residuals of a least-squares fit with an intercept sum to zero and are uncorrelated with every feature.
  • PCA projects the data onto the directions of largest variance, and the discarded part is orthogonal to the kept directions (PCA).
  • Gram-Schmidt and QR decompositions build orthogonal directions, which is how many libraries solve least squares in a numerically stable way.

For a set of columns collected in a matrix X, the projection matrix is P = X (XᵀX)⁻¹ Xᵀ, and P y is the least-squares prediction.

Python Projection onto a direction, and regression as a projection

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The residual is exactly perpendicular to the direction a, and the least-squares fit is exactly the projection P y: the residual is orthogonal to the constant column (so the residuals sum to zero) and to each feature. Projecting twice gives the same result as projecting once, which is the defining property of any projection matrix.

A few more terms you will meet

  • Scalar, vector, matrix, tensor: a number, a 1-D list, a 2-D table, and the general n-dimensional array (a batch of color images is a 4-D tensor).
  • Partial derivative: the slope with respect to one variable while the others stay fixed. The gradient is the vector of all of them.
  • Convex function: a bowl with a single lowest point. Linear and logistic regression losses are convex, so gradient descent finds the best answer; neural network losses are not.
  • Orthogonal: at right angles, dot product zero. PCA axes are orthogonal.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What does a cosine similarity of 0 between two vectors mean?

    The cosine is the dot product divided by both lengths. A dot product of 0 means the vectors are at right angles. Identical direction gives 1 and opposite direction gives -1.
  2. 2. In X @ w, X has shape (200, 3). What shape must w have, and what shape is the result?

    The last dimension of X (3 features) must match the first dimension of w. Each of the 200 rows is dotted with w, so the result has one number per row.
  3. 3. Why does gradient descent subtract the gradient instead of adding it?

    The gradient points uphill. Moving against it lowers the loss, which is the goal of training.
  4. 4. What is the chain rule used for in machine learning?

    A model feeds the output of one step into the next. The chain rule gives the derivative of the whole chain, and backpropagation applies it layer by layer.

Summary

  • A vector is one example, a matrix is a table of examples, and a prediction is a dot product weights · features + bias.
  • X @ w predicts for every row at once; check shapes first when anything fails.
  • Least squares solves XᵀX w = Xᵀy; use np.linalg.solve or lstsq, not an explicit inverse.
  • Eigenvectors of the covariance matrix are the axes of the data cloud, which is the idea behind PCA; SVD computes them stably.
  • The derivative is a slope, the gradient is the vector of slopes, and gradient descent steps against it: w ← w − lr × gradient.
  • The chain rule multiplies slopes along nested steps; check hand-written gradients against a numerical derivative.

Next, the other half of the foundations: Probability and Statistics for ML.