Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 62 · Eigenvalues and Decompositions

The Covariance Matrix

Learn the covariance matrix: variances on the diagonal, covariances off it, 2x2 and 3x3 examples by hand, positive semi-definiteness and its link to PCA.

A table of measurements, say the height, weight and age of 200 people, has more structure than any single column shows. Tall people tend to weigh more; age may not matter much. The covariance matrix captures all of this in one small square matrix: how much each feature varies on its own, and how each pair of features varies together.

It is the bridge between data and the linear algebra of this course. The covariance matrix is symmetric and positive semi-definite, so the spectral theorem gives it real eigenvalues and orthogonal eigenvectors, and those eigenvectors are the principal components used in PCA.

Variance and covariance

For one feature x with n observations and mean x̄, the variance is the average squared distance from the mean (dividing by n - 1 is the usual unbiased "sample" version):

TEXT
var(x) = Σ (x_i - x̄)² / (n - 1)

For two features x and y, the covariance multiplies the two deviations together instead of squaring one:

TEXT
cov(x, y) = Σ (x_i - x̄)(y_i - ȳ) / (n - 1)

Positive covariance means the features rise together, negative means one rises as the other falls, and near zero means no straight-line relationship. The variance is a covariance of a feature with itself.

The matrix

Collect d features into an n × d data matrix X (one row per observation). Subtract each column's mean to get the centered matrix X_c. Then

TEXT
Σ = X_cᵀ X_c / (n - 1)

The entry Σ[i, j] = cov(feature i, feature j). So the diagonal holds the variances and the matrix is symmetric because cov(x, y) = cov(y, x). Its trace is the total variance of the data.

A 2 × 2 example by hand. Take 4 observations of (x, y): (1, 2), (2, 3), (3, 5), (4, 6). The means are x̄ = 2.5 and ȳ = 4. The deviations are x: -1.5, -0.5, 0.5, 1.5 and y: -2, -1, 1, 2.

  • var(x) = (2.25 + 0.25 + 0.25 + 2.25)/3 = 5/3
  • var(y) = (4 + 1 + 1 + 4)/3 = 10/3
  • cov(x, y) = (3 + 0.5 + 0.5 + 3)/3 = 7/3

So Σ = [[5/3, 7/3], [7/3, 10/3]], about [[1.667, 2.333], [2.333, 3.333]].

The correlation matrix rescales covariance by the standard deviations: corr(x, y) = cov(x, y) / (σ_x σ_y), a number between -1 and 1, with ones on the diagonal. Here it is (7/3) / sqrt(5/3 · 10/3) = 0.9899, a very strong positive relationship.

Python Covariance by hand, with np.cov, and the correlation matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the height-weight covariance is large, the height-noise covariance is close to zero, and every eigenvalue is nonnegative.

Why it is positive semi-definite

For any vector w, the number wᵀ Σ w is the variance of the combination w₁·feature₁ + w₂·feature₂ + ... of the features. A variance can never be negative, so wᵀ Σ w ≥ 0 for every w. That is the definition of positive semi-definite, and it means all eigenvalues of Σ are ≥ 0. It is strictly positive definite (and invertible) unless some combination of the features is constant, for example when one feature is a copy of another.

Eigenvectors are the principal axes

The eigen-decomposition Σ = Q Λ Qᵀ has a direct data meaning:

  • Each eigenvector is a direction in feature space. The one with the largest eigenvalue is the direction along which the data vary the most.
  • Each eigenvalue is the variance of the data along its eigenvector.
  • Projecting the centered data onto the eigenvectors gives new features that are uncorrelated: their covariance matrix is the diagonal matrix Λ.

Keeping only the top few eigenvectors is PCA. You can also whiten data (make the covariance the identity) by dividing each rotated coordinate by sqrt(λ), or by multiplying with the inverse of a Cholesky factor.

Python Principal axes, uncorrelated coordinates and whitening

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recovered first axis lies at about 30 degrees, matching the rotation we applied (an eigenvector's sign is arbitrary, so we reduce the angle modulo 180), and its variance is near 3² = 9. After rotating, the covariance matrix is diagonal; after whitening, it is the identity.

Mind the shape conventions. np.cov expects variables in rows by default, whereas data matrices usually have variables in columns. Pass rowvar=False or transpose, otherwise you get an n × n matrix instead of d × d. Also divide by n - 1 (ddof=1) for a sample, or by n when the data is the entire population.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What do the diagonal entries of a covariance matrix represent?

    Entry (i, i) is the covariance of feature i with itself, which is its variance. Ones on the diagonal belong to the correlation matrix.
  2. 2. Which statement about a covariance matrix is always true?

    cov(x, y) = cov(y, x) gives symmetry, and w^T Sigma w is the variance of a linear combination, which cannot be negative.
  3. 3. For the data (1,2), (2,3), (3,5), (4,6), cov(x, y) is positive. What does that tell you?

    Positive covariance means that when x is above its mean, y tends to be above its mean as well.
  4. 4. In PCA, what is the eigenvalue belonging to a principal component?

    The eigenvalue of the covariance matrix is the variance of the data projected on the matching eigenvector.
  5. 5. You call np.cov(X) on a 500 x 4 data matrix (500 samples, 4 features) and get a 500 x 500 result. What went wrong?

    By default np.cov assumes each row is a variable. Pass rowvar=False (or X.T) for samples in rows.

Summary

  • The covariance matrix Σ = X_cᵀ X_c / (n - 1) has variances on its diagonal and covariances off it; it is symmetric.
  • The correlation matrix rescales covariances to the range -1 to 1.
  • Σ is positive semi-definite because wᵀΣw is the variance of a combination of features.
  • Eigenvectors of Σ are the principal axes of the data and eigenvalues are the variances along them; rotating onto them decorrelates the features.
  • Watch the rowvar convention in np.cov.

Next: Complex Matrices: Conjugate Transpose, Hermitian and Unitary, for matrices with complex entries.