Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 62 ยท Eigenvalues and Decompositions

Symmetric Matrices and the Spectral Theorem

Why symmetric matrices have real eigenvalues and orthogonal eigenvectors, how to use np.linalg.eigh, and how quadratic forms and positive definiteness follow.

Symmetric matrices, the ones equal to their own transpose (A = A^T), are the best-behaved matrices in linear algebra. You have already seen how often they appear: A^T A is always symmetric (Transpose and Symmetric Matrices), and so are covariance matrices, distance-like tables and the second-derivative matrices used in optimization.

Their eigenvalues and eigenvectors follow rules that general matrices do not obey. They are always real, the eigenvectors can be chosen perpendicular, and the matrix is always diagonalizable, so the troubles of defective matrices never happen. This result is the spectral theorem. It leads straight to quadratic forms and to the idea of a positive definite matrix, which tells you whether an optimization problem has a bowl-shaped minimum.

What the spectral theorem promises

Real eigenvalues No complex numbers ever appear.
Orthogonal vectors Eigenvectors can be chosen at right angles.
Always diagonalizable A = Q D Q^T with Q orthogonal.

Use np.linalg.eigh, not eig, for symmetric input.

1 / 2

The spectral theorem

For a real symmetric matrix A there is an orthogonal matrix Q (its columns are perpendicular unit vectors, so Q^T Q = I) and a diagonal matrix D of real numbers with

A = Q D Q^T

Compare this with the general P D P^-1. For symmetric matrices the awkward inverse P^-1 is replaced by the transpose Q^T, which costs nothing to compute and does not amplify rounding errors. Geometrically: rotate (or reflect) to the eigenvector axes, scale each axis, rotate back.

Why are the eigenvalues real and the vectors perpendicular? Here is the key step for the second claim. If A v1 = l1 v1 and A v2 = l2 v2 with l1 different from l2, then l1 (v1 . v2) = (A v1) . v2 = v1 . (A^T v2) = v1 . (A v2) = l2 (v1 . v2). The middle equality uses A = A^T. So (l1 - l2)(v1 . v2) = 0, which forces v1 . v2 = 0.

NumPy has a dedicated routine, np.linalg.eigh. It assumes the input is symmetric (it only reads one triangle), returns real eigenvalues in ascending order, and returns orthonormal eigenvectors as columns. The general np.linalg.eig makes none of these promises, even when you hand it a symmetric matrix.

Python eigh gives Q and D

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two lines are a contrast. A rotation by 90 degrees is not symmetric, no real direction stays fixed, and eig returns the complex pair 0+1j and 0-1j. Symmetry rules this out.

A sum of rank-one pieces

Multiplying out Q D Q^T column by column gives another way to read the theorem:

A = l1 q1 q1^T + l2 q2 q2^T + ... + ln qn qn^T

Each q q^T is a rank-one matrix that projects onto the line through q (Orthogonality and Projections). So a symmetric matrix is a weighted sum of projections onto perpendicular directions, weighted by the eigenvalues. Dropping the small terms gives a simpler matrix that is still close to A, which is the idea you will see again, in its most general form, in the SVD.

The decomposition also gives you functions of a matrix for free. Because A^k = Q D^k Q^T, the inverse is Q D^-1 Q^T (invert every eigenvalue) and a square root is Q sqrt(D) Q^T, as long as the eigenvalues are positive.

Python Rebuild a matrix from rank-one pieces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quadratic forms

A quadratic form takes a vector x and returns one number, x^T A x. For a 2 x 2 symmetric matrix [[a, b], [b, c]] it expands to a x1^2 + 2 b x1 x2 + c x2^2: every term has degree two. Quadratic forms describe energy, variance and the local shape of a function near a critical point.

The spectral theorem makes the shape easy to read. Write x in the eigenvector basis, with coordinates y = Q^T x. Then

x^T A x = l1 y1^2 + l2 y2^2 + ... + ln yn^2

There are no cross terms in these coordinates. If all eigenvalues are positive, the form is a bowl; if some are positive and some negative, it is a saddle. Restricted to unit vectors (||x|| = 1), the form is largest at the eigenvector of the largest eigenvalue and smallest at the eigenvector of the smallest, with exactly those eigenvalues as the extreme values. This is called the Rayleigh quotient idea, and it is why PCA uses the top eigenvector of a covariance matrix to find the direction of greatest variance.

Python Extreme values of a quadratic form

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Positive definite matrices

A symmetric matrix A is positive definite if x^T A x > 0 for every nonzero vector x, and positive semidefinite if x^T A x >= 0. By the formula above, this is the same as saying all eigenvalues are positive (or non-negative). Positive definite matrices behave like positive numbers: they have a square root, they are invertible, and the quadratic form is a bowl with one minimum. In optimization, the matrix of second derivatives being positive definite is what certifies a local minimum.

There are three practical tests:

  1. Eigenvalues: np.linalg.eigvalsh(A).min() > 0.
  2. Cholesky: in desktop NumPy, np.linalg.cholesky(A) succeeds only for positive definite matrices and raises LinAlgError otherwise, which makes it the cheapest test. The in-browser NumPy on this site does not reliably raise, so the example below uses eigenvalues.
  3. Gram form: any matrix of the shape B^T B is positive semidefinite, because x^T B^T B x = ||B x||^2 >= 0. It is positive definite when the columns of B are independent.
Python Testing positive definiteness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dependent-column example shows why you should compare to a small tolerance instead of exactly zero: the true smallest eigenvalue is 0, but floating-point arithmetic returns a tiny number whose sign can vary.

np.linalg.eigh trusts you. If you pass a matrix that is not symmetric, it silently uses only the lower triangle and returns an answer for a different matrix. Check np.allclose(A, A.T) first, or build (A + A.T) / 2 when tiny asymmetries come from rounding.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement is guaranteed for a real symmetric matrix?

    That is the spectral theorem. Positive eigenvalues or invertibility need extra conditions.
  2. 2. In A = Q D Q^T, what replaces the P^-1 of ordinary diagonalization?

    An orthogonal matrix satisfies Q^T Q = I, so its transpose is its inverse.
  3. 3. Which test tells you that a symmetric matrix is positive definite?

    Positive definite means x^T A x > 0 for all nonzero x, which is equivalent to every eigenvalue being positive.
  4. 4. On unit vectors, what is the largest value of x^T A x for symmetric A?

    The maximum is reached at the eigenvector of the largest eigenvalue, and equals that eigenvalue.

Summary

  • A real symmetric matrix has real eigenvalues, perpendicular eigenvectors and factors as A = Q D Q^T with Q orthogonal (the spectral theorem).
  • Use np.linalg.eigh for symmetric input: real values in ascending order, orthonormal eigenvector columns.
  • A is a sum of rank-one projections weighted by its eigenvalues, which makes inverses and square roots easy.
  • A quadratic form x^T A x has no cross terms in eigenvector coordinates; its extreme values on unit vectors are the largest and smallest eigenvalues.
  • Positive definite means all eigenvalues are positive; test with eigvalsh (or np.linalg.cholesky on desktop NumPy). Matrices B^T B are always at least semidefinite.

Next, extend these ideas to every matrix with the Singular Value Decomposition.