Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 76 ยท Math Foundations

Linear Algebra for Data Science

Learn the linear algebra data scientists use: vectors as rows of data, dot products and similarity, matrices as tables, least squares, eigenvectors and SVD, with NumPy.

Linear algebra sounds like a school subject you left behind, but a data scientist uses it every day without noticing. A table of data is a matrix. A prediction is usually a dot product. Finding the best line through points is a matrix equation. Reducing a hundred columns to two for a chart is an eigenvector calculation. If you understand a few ideas with pictures and NumPy, much of machine learning stops being magic.

You do not need to prove theorems. This lesson explains what the objects mean for data and how to compute with them. The NumPy course goes deeper into the array mechanics, and the SciPy course covers the numerical algorithms.

Vectors: one row of data is a point

A vector is an ordered list of numbers. In data science, one observation is a vector: a house with [area, bedrooms, age] is the point [120, 3, 15] in three-dimensional space. A whole column of data is also a vector, with one entry per observation.

Two basic operations matter:

  • Adding and scaling vectors works entry by entry. The average of a group of customers is a vector sum divided by the count.
  • Length (the norm) says how big a vector is. The straight-line distance between two observations is the norm of their difference. That is the "distance" used by nearest-neighbour and clustering algorithms.
Python Vectors, distance and length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

House a is closer to c than to b, so a nearest-neighbour model would call them similar. One warning: distance depends on the units of each column. Here, the area in square metres dominates the distance, which is why columns are usually scaled first (see Encoding and Scaling).

The dot product: weighted sums and similarity

The dot product multiplies two vectors entry by entry and adds the results: a . b = a1*b1 + a2*b2 + .... It appears everywhere:

  • A weighted sum is a dot product: a price estimate is weights . features.
  • Similarity: divide the dot product by both lengths to get the cosine similarity, a number between -1 and 1 that compares direction and ignores size. It is the standard way to compare documents, products and user tastes.
Python Dot product and cosine similarity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Document 2 is twice as long as document 1 but points in the same direction, so their cosine is close to 1. The gardening document is orthogonal (at 90 degrees) and scores 0. The @ operator is Python's matrix multiplication, and for two vectors it is the dot product.

Matrices: a table of data

A matrix is a rectangular grid of numbers. A data table with n rows (observations) and p columns (features) is a matrix X of shape (n, p). Multiplying it by a vector of weights gives a prediction for every row at once:

predictions = X @ w

That one line is the core of linear regression, and also the first layer of every neural network (a stack of dot products, one per neuron, followed by a non-linear step). The shape rule is (n, p) @ (p,) -> (n,): the inner sizes must match.

flowchart LR
    X["X: n rows x p features"] -->|"@"| W["w: p weights"]
    W --> Y["predictions: n numbers"]

A model's predictions for every observation come from one matrix-vector product.

Solving for the best weights: least squares

If you know the answers y and want the weights w that make X @ w as close to y as possible, you are solving a least-squares problem. Add a column of ones to X so that the model has an intercept. NumPy's np.linalg.lstsq finds the best w:

Python Linear regression as linear algebra

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weights come out close to the rule we used to generate the data (2.5 per square metre and 12 per bedroom; the noise moves the estimates a little, most visibly the bedroom weight). The rank tells you how many columns carry independent information. If two features are perfectly correlated (say, area in square feet and in square metres), the rank drops below p, the equations have no unique answer, and the weights become unstable. This is called multicollinearity, and it is one of the best reasons to understand rank.

Eigenvectors and PCA: finding the main directions

Data cloud pictures often look like a stretched ellipse. The covariance matrix describes this shape, and its eigenvectors point along the ellipse's axes, while the eigenvalues say how much the data spreads along each one. Principal component analysis (PCA) keeps the directions with the largest eigenvalues, so that you can describe the data with fewer columns. See Dimensionality Reduction for the modelling side.

Python PCA from eigenvectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The red arrow points along the long axis of the cloud and carries most of the variance; the green arrow, perpendicular to it, carries the rest. Replacing two columns by the single scores column keeps most of the information. With 100 columns the idea is the same: keep the few directions with the biggest eigenvalues.

SVD: the Swiss army knife

Any matrix can be factored as X = U @ diag(s) @ Vt, called the singular value decomposition (SVD). The singular values s are sorted from largest to smallest and say how important each component is. Keeping only the top k of them gives the best possible rank-k approximation of X, which is the basis of compression, noise removal and recommender systems.

Python Low-rank approximation with SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only two singular values are large, because only two hidden tastes generated the data. A rank-2 approximation is already almost perfect (the remaining error is the noise), and adding a third component gains little. That drop is how you decide how many components to keep.

Where this appears in the rest of the course

Idea Used in
X @ w Linear and logistic regression, neural networks
Distance and cosine similarity Nearest neighbours, clustering, text search
Least squares and rank Regression, multicollinearity
Eigenvectors, SVD PCA, recommenders, image compression

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A table has 500 rows and 8 feature columns. What is the shape of the weight vector for X @ w?

    The inner dimensions must match: (500, 8) @ (8,) gives 500 predictions. So w has one weight per feature.
  2. 2. What does cosine similarity compare?

    Dividing the dot product by both lengths removes size. Two documents of different lengths can still point in the same direction.
  3. 3. Why add a column of ones to X in least squares?

    The weight of the constant column is the intercept, the prediction when all features are zero.
  4. 4. What do the eigenvectors of the covariance matrix point along?

    The eigenvectors are the axes of the data cloud. Eigenvalues tell how much variance lies along each axis, which is the basis of PCA.
  5. 5. What does a rank lower than the number of columns warn you about?

    A rank deficit means the columns carry less independent information than their count, which makes least squares unstable (multicollinearity).

Summary

  • A row of data is a vector (a point); distance is the norm of a difference; scale features before using distances.
  • The dot product is a weighted sum; dividing by the lengths gives cosine similarity.
  • A data table is a matrix; X @ w predicts every row at once.
  • Least squares (np.linalg.lstsq) finds the weights that best fit y; rank warns about redundant columns.
  • Eigenvectors of the covariance matrix are the principal directions (PCA); SVD gives the best low-rank approximation.

Next: Calculus and Optimization for Data Science explains how models learn their weights by following slopes.