Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 49 ยท Projects

Project: Linear Regression from Scratch

Build linear regression with NumPy only. Generate data, solve the normal equation, train with vectorized gradient descent, plot the learning curve, and compare with lstsq.

Linear regression is the "hello world" of machine learning: given some inputs, predict a number as a weighted sum of them. Libraries hide the details behind a single fit() call, but the whole algorithm fits in about ten lines of NumPy. Building it yourself is the best way to see why matrix multiplication, broadcasting and solving linear systems matter.

In this project you will invent a small data set of apartment prices, fit a model in two different ways (the normal equation and gradient descent), watch the training error shrink on a learning-curve plot, measure quality with R squared, and check your answer against np.linalg.lstsq. Everything is deterministic thanks to a seeded random generator.

The model in matrix form

For one apartment, the prediction is price = w0 + w1 * size + w2 * rooms. The constant w0 is the intercept. To handle it without special cases, we add a column of ones to the data. Then, for all apartments at once, the predictions are a single matrix-vector product:

TEXT
predictions = X @ w

X has one row per apartment and one column per feature (plus the column of ones), so its shape is (n_samples, n_features + 1). The weight vector w has n_features + 1 entries. Training means finding the w that makes the predictions close to the real prices, where "close" is measured by the mean squared error (MSE): the average of the squared differences.

flowchart LR
    D["Data X, y"] --> M["Model<br/>predict = X @ w"]
    M --> L["Loss<br/>mean squared error"]
    L --> G["Gradient<br/>2/n * X.T @ error"]
    G --> U["Update<br/>w = w - lr * gradient"]
    U --> M

Gradient descent loops through predict, measure, compute the gradient and update until the loss stops improving.

Step 1: synthetic data

We create data from a known formula plus random noise, so we can tell whether our fit recovers the truth. Real prices would be 20 + 1.5 * size + 8 * rooms, plus noise with a standard deviation of 10.

Python Generate data and build the design matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

np.column_stack glues the three 1D arrays together as columns (see Joining and Splitting). Keep an eye on shapes: X is (100, 3) and price is (100,). Most beginner bugs in this kind of code are shape bugs, and printing shapes early saves time.

Step 2: the normal equation

For least squares there is a closed-form answer. Setting the gradient of the loss to zero gives the normal equations:

TEXT
(X.T @ X) @ w = X.T @ y

This is an ordinary linear system A @ w = b, so we solve it with np.linalg.solve. We do not compute an inverse, because solving is faster and more accurate (see Matrix Operations).

We also need a way to score a fit. R squared compares our squared error with the error of the dumbest possible model, one that always predicts the mean price:

TEXT
R2 = 1 - (sum of squared residuals) / (total sum of squares)

A value of 1 means a perfect fit, 0 means no better than predicting the mean.

Python Fit with the normal equation and score with R squared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted size weight lands very close to the true 1.5, and the rooms weight (about 7) is in the neighborhood of the true 8. They are not exact, because the noise in 100 samples blurs the picture. The RMSE (root mean squared error) of about 10 matches the noise we added: the model cannot beat the randomness we put into the data.

The normal equation is elegant, but it squares the condition number of the problem, and its cost grows with the cube of the number of features. For millions of rows it is cheap (the expensive part is X.T @ X), but for thousands of features or badly scaled columns people use QR/SVD-based solvers or gradient descent.

Step 3: gradient descent, fully vectorized

Gradient descent finds the weights by walking downhill on the loss surface. For the MSE loss, the gradient with respect to all weights at once is:

TEXT
gradient = (2 / n) * X.T @ (X @ w - y)

One matrix product computes the predictions, one subtraction gives the errors, and a second matrix product turns the errors into a gradient for every weight. No loop over samples or features is needed, only a loop over training steps.

Before training, we standardize the features: subtract each column's mean and divide by its standard deviation. Without this, the "size" column (values around 30 to 120) dominates the "rooms" column (1 to 5), the loss surface becomes a narrow valley, and gradient descent needs a tiny learning rate and many steps. Standardizing is a broadcasting one-liner: (features - mean) / std where mean and std have shape (2,) and broadcast across the 100 rows.

Python Train with gradient descent and plot the learning curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls steeply at first and then flattens near the noise floor of about 100 (that is the noise variance, 10 squared). The log-scale plot makes it easy to see when the curve has stopped improving. After converting the weights back to the original units (slopes = w[1:] / std and the matching intercept), they match the normal-equation answer.

If the learning rate is too small, training is slow; too large and the loss bounces or explodes. Learning rate is the one knob you must tune by hand.

Step 4: compare with lstsq and break the learning rate

np.linalg.lstsq is NumPy's production-grade least-squares solver. It uses an SVD-based method and copes with nearly singular problems. It is the reference answer for our hand-made versions. In the same example we deliberately set a too-large learning rate to see divergence.

Python Check against lstsq and try different learning rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three methods agree. The normal equation and lstsq are direct solvers and give the same weights to many decimal places. The learning-rate experiment shows the typical pattern: 0.001 is still far from converged after 100 steps, 0.1 has converged, and a rate above the stability limit (here around 1.0 for standardized data) blows up to astronomically large losses.

Where to go from here

  • Regularization: add alpha * w terms to the gradient (ridge regression) and watch the weights shrink.
  • Polynomial features: add size ** 2 as a column; the model is still linear in the weights.
  • Train/test split: shuffle with rng.permutation, fit on 80 percent, and compute R squared on the rest to detect overfitting.
  • Mini-batches: use Z[batch_indices] inside the loop, which is how large models train.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we add a column of ones to the feature matrix?

    Multiplying a column of ones by its weight adds the same constant to every prediction. That weight is the intercept, and the model stays a single matrix product X @ w.
  2. 2. What is the shape of the gradient (2 / n) * Z.T @ errors when Z has shape (100, 3) and errors has shape (100,)?

    Z.T has shape (3, 100). Multiplying by a vector of length 100 gives a vector of length 3: one gradient entry per weight.
  3. 3. What does an R squared of 0 mean?

    R squared compares your squared error with the error of the constant-mean predictor. 1 is perfect, 0 means no improvement over the mean, and negative values mean worse than the mean.
  4. 4. Why standardize features before gradient descent?

    When one feature has much larger values than another, the loss surface becomes a narrow valley and gradient descent zigzags. Standardized features give a rounder surface and faster, more stable training.

Summary

  • A linear model is X @ w, with a column of ones in X for the intercept.
  • The normal equation (X.T @ X) w = X.T @ y is solved with np.linalg.solve, never with an explicit inverse.
  • Vectorized gradient descent uses gradient = (2 / n) * X.T @ (X @ w - y) and a loop only over steps.
  • Standardize features with broadcasting; convert weights back to original units afterwards.
  • R squared and RMSE judge the fit; np.linalg.lstsq is the trusted reference.

Next project: A Tiny Neural Network.