Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 49 ยท Linear Algebra

Eigenvalues, SVD and Decompositions

Learn eigenvalues and eigenvectors, SVD, QR and Cholesky decompositions in NumPy, with low-rank approximation, PCA in a few lines and image compression.

A decomposition splits a matrix into simpler pieces whose product gives the original back. Like factoring 12 into 3 x 4, the pieces reveal structure that the original numbers hide: directions a matrix stretches, which parts carry the most information, or a faster way to solve equations.

This lesson tours the four decompositions you will meet most often in numpy.linalg: eigen-decomposition (eig, eigh), singular value decomposition (svd), QR (qr) and Cholesky (cholesky). Then it uses the SVD for two practical jobs, compressing an image and running PCA. You need to be comfortable with @ and transposes from the matrices lesson.

Eigenvalues and eigenvectors

Most vectors change direction when a matrix multiplies them. An eigenvector of A is a special vector that only gets stretched: A @ v = ฮป * v. The stretch factor ฮป (lambda) is its eigenvalue.

np.linalg.eig(A) returns the eigenvalues and a matrix whose columns are the matching eigenvectors. Note the columns: vectors[:, i] pairs with values[i].

Python Eigenvalues and eigenvectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trace and determinant links at the end are two handy sanity checks for any square matrix.

flowchart LR
    V["vector v"] -->|"multiply by A"| W["A v"]
    V2["eigenvector v"] -->|"multiply by A"| W2["ฮป v<br/>(same direction, scaled)"]

An ordinary vector changes direction under A; an eigenvector only changes length.

eigh for symmetric matrices

Covariance matrices and many matrices in physics and statistics are symmetric (equal to their transpose). For these, use np.linalg.eigh. It is faster, always returns real numbers, returns eigenvalues in ascending order and gives perpendicular (orthogonal) eigenvectors.

Python eigh for symmetric matrices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use eig only for general, non-symmetric matrices. Its results may be complex numbers, as for a rotation matrix, which has no real direction that stays fixed.

Singular value decomposition (SVD)

The SVD works for any matrix, including non-square ones. It writes A = U @ diag(S) @ Vt, where U and Vt are rotations (orthogonal matrices) and S holds non-negative singular values in descending order. Read it as: rotate, stretch along the axes, rotate again.

By default np.linalg.svd(A) returns full square U and Vt. Pass full_matrices=False for the compact form, which is almost always what you want.

Python SVD and rebuilding the matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

flowchart LR
    A["A (m x n)"] --> D["np.linalg.svd"]
    D --> U["U (m x k)<br/>rotation"]
    D --> S["S (k,)<br/>singular values"]
    D --> VT["Vt (k x n)<br/>rotation"]

The compact SVD splits a matrix into two rotations and a list of stretches; k is min(m, n).

Low-rank approximation

The SVD is the best way to simplify a matrix. Keep only the largest k singular values and drop the rest: U[:, :k] @ diag(S[:k]) @ Vt[:k]. This rank-k approximation is provably the closest possible matrix of rank k.

Python Low-rank approximation error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two singular values stand tall and the rest are tiny, so rank 2 captures almost everything. Reading a "scree" of singular values like this is the standard way to decide how many dimensions matter.

Image compression

A grayscale image is a matrix, so low-rank approximation compresses it. Storing a rank-k version needs k * (m + n + 1) numbers instead of m * n. Here we build a small synthetic image (no files needed) and compare ranks.

Python Compressing an image with SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

PCA in a few lines

Principal component analysis finds the directions along which data varies most. With the SVD it is short: center the data, decompose, and project onto the first rows of Vt.

Python PCA with the SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first component explains nearly all the variance, which tells you the data is almost one-dimensional.

QR and Cholesky

Two more decompositions are mainly used as building blocks:

  • QR: Q, R = np.linalg.qr(A) where Q has orthonormal columns and R is upper triangular. It is the stable way to solve least-squares problems and the basis of many algorithms.
  • Cholesky: L = np.linalg.cholesky(A) for a symmetric, positive-definite A, giving A = L @ L.T with L lower triangular. It is about twice as fast as other methods and is used to sample correlated random numbers. If the matrix is not positive definite, NumPy raises LinAlgError.
Python QR and Cholesky

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In np.linalg.eig(A), how are the eigenvectors returned?

    vectors[:, i] is the eigenvector for values[i]. Mixing up rows and columns is a very common mistake.
  2. 2. Which function is best for a symmetric matrix such as a covariance matrix?

    eigh is faster, returns real eigenvalues in ascending order and gives orthogonal eigenvectors for symmetric input.
  3. 3. How do you build the best rank-k approximation of A from its SVD?

    U[:, :k] @ diag(S[:k]) @ Vt[:k] is the closest rank-k matrix, since singular values come sorted from largest to smallest.
  4. 4. What does np.linalg.cholesky raise for a matrix that is not positive definite?

    Cholesky requires a symmetric positive-definite matrix and raises numpy.linalg.LinAlgError otherwise.

Summary

  • np.linalg.eig gives eigenvalues and eigenvectors (as columns); use eigh for symmetric matrices.
  • trace(A) equals the sum of eigenvalues and det(A) equals their product.
  • np.linalg.svd(A, full_matrices=False) factors any matrix as U @ diag(S) @ Vt, with singular values sorted descending.
  • Keeping the top k singular values gives the best rank-k approximation, which powers image compression and noise reduction.
  • PCA is centering the data, taking the SVD and projecting onto the rows of Vt.
  • qr gives orthonormal Q and triangular R; cholesky factors a positive-definite matrix as L @ L.T.

Next, write all of these products compactly with Einstein Summation with einsum.