Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 38 ยท Linear Algebra

Eigenvalues and Singular Values

Compute eigenvalues with eig and eigh, factor matrices with svd, build low-rank image approximations, run PCA from scratch and use the matrix exponential expm.

Eigenvalues and singular values reveal what a matrix really does. An eigenvector is a direction a matrix only stretches, never turns. Singular values measure how strongly a matrix stretches space along its principal axes. Together they power vibration analysis, Google's original PageRank, image compression, noise reduction and principal component analysis (PCA).

This lesson uses eig, eigh and svd from scipy.linalg, shows how to compress an image with a low-rank approximation, builds PCA from scratch and ends with the matrix exponential expm. It builds on the decompositions lesson.

Eigenvalues and eigenvectors

A number lambda and a non-zero vector v form an eigenpair of A when A v = lambda v. SciPy gives you two main functions:

  • linalg.eigh(A) for symmetric (or complex Hermitian) matrices. The eigenvalues are real, returned in ascending order, and the eigenvectors are orthogonal. It is faster and more accurate. Use it whenever it applies.
  • linalg.eig(A) for general square matrices. It may return complex numbers and the order is not guaranteed, so sort if order matters.

The eigenvectors come back as the columns of a matrix V, so A @ V equals V * w (each column scaled by its eigenvalue).

Python eigh, eig and complex eigenvalues

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rotation matrix turns every vector, so it has no real eigenvector. Its eigenvalues are the complex pair i and -i. That is not a bug, just math: complex results are normal for non-symmetric matrices.

Tip: Use eigvals or eigvalsh when you only need the values; skipping the eigenvectors is cheaper.

The singular value decomposition

The SVD works for any matrix, even rectangular ones: A = U S V^T, where U and V have orthonormal columns and S is a diagonal matrix of non-negative singular values sorted from largest to smallest. SciPy returns the singular values as a 1-D array s and the transposed factor Vt.

By default full_matrices=True builds square U and Vt. Pass full_matrices=False for the compact form, which is nearly always what you need.

flowchart LR
    A["A<br/>m by n"] --> U["U<br/>m by k"]
    A --> S["s<br/>k values, descending"]
    A --> V["Vt<br/>k by n"]

In the compact SVD, k is min(m, n). Multiply back with U times diag(s) times Vt to recover A.

Python SVD of a rectangular matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Counting singular values above a small tolerance gives the numerical rank, the most trustworthy way to find it. The largest singular value equals the spectral norm, and the ratio of largest to smallest is the condition number from the first linear algebra lesson.

Low-rank approximation and image compression

Here is the SVD's most useful property. If you keep only the first k singular values and their vectors, you get the best possible rank-k approximation of the matrix (the Eckart-Young theorem). That matrix needs only k * (m + n + 1) numbers instead of m * n.

We build a small synthetic 60-by-60 "image", a pair of smooth blobs plus a ramp and a touch of noise, then compare rank 1, 3 and 10.

Python Compress an image with a low-rank SVD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rank 3 stores about a tenth of the numbers (363 of 3,600) yet keeps over 99.7% of the "energy" (the sum of squared singular values) and is visually almost identical. Beyond rank 3 the error barely improves, because what remains is mostly the random noise we added: low-rank truncation is also a denoiser. Real photos need a higher rank, but the same trade-off between size and error applies.

PCA from scratch

Principal component analysis finds the directions along which data varies most. The recipe is short:

  1. Subtract the column means (center the data).
  2. Take the SVD of the centered matrix.
  3. The rows of Vt are the principal directions; s**2 / (n - 1) are the variances along them; projecting with Xc @ Vt[:k].T gives the scores.
Python PCA with the SVD and with eigh

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Our three features come from one hidden factor t (the first two) plus an independent noisy one, so the first component captures more than 90% of the variance and points mostly along the first two features. Using the SVD is preferred to forming the covariance matrix because squaring the data (X^T X) squares the condition number. The eigen route is shown only to confirm both agree.

The matrix exponential

For a matrix A, expm(A) is the matrix exponential, defined by the series I + A + A^2/2! + .... It is not the element-wise np.exp. It solves linear differential equations: y' = A y with y(0) = y0 has the exact solution y(t) = expm(A t) @ y0. Rotations come from the exponential of a skew-symmetric matrix. Siblings include sqrtm, logm and fractional_matrix_power.

Python expm for rotations and linear ODEs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The decaying values match exp(-t) and exp(-2t), the exact solution for this diagonal system, with no step-by-step solver needed. For nonlinear equations you need the solvers of the integration chapter.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. When should you use linalg.eigh instead of linalg.eig?

    eigh exploits symmetry to return real eigenvalues in ascending order with orthogonal eigenvectors, faster and more accurately than eig.
  2. 2. linalg.svd(A) returns U, s, Vt. What is s?

    SciPy returns the singular values as a 1-D array sorted largest first. Build the diagonal matrix yourself if you need it.
  3. 3. What do you get by keeping only the first k singular values and vectors?

    By the Eckart-Young theorem, truncating the SVD gives the closest possible rank-k matrix in both the Frobenius and spectral norms.
  4. 4. Which statement about expm(A) is correct?

    The matrix exponential is defined by a power series of matrices and is not element-wise. It gives the exact solution of linear constant-coefficient ODEs.
  5. 5. Why does PCA prefer the SVD of the centered data over the eigen-decomposition of X^T X?

    Squaring the data matrix squares its condition number, so small singular values lose accuracy. The SVD avoids that step.

Summary

  • eigh for symmetric matrices, eig for general ones (expect complex results); eigvals and eigvalsh skip the vectors.
  • The SVD A = U S V^T exists for every matrix; SciPy returns U, s, Vt with s sorted descending.
  • Truncating the SVD gives the best low-rank approximation, useful for compression and denoising.
  • PCA is center, SVD, project; s**2 / (n - 1) are the variances.
  • expm is the matrix exponential and solves linear ODEs exactly.

Next: Least Squares and Pseudo-inverses.