Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 62 ยท Projects

Project: Principal Component Analysis

Build principal component analysis from scratch with NumPy: covariance matrix, eigenvectors, SVD, projection, explained variance, and a scikit-learn check.

Principal component analysis (PCA) is the most popular application of the eigenvalue and SVD ideas you have just learned. Given a table of measurements with many correlated columns, PCA finds a new set of perpendicular axes, ordered by how much the data varies along them. Keeping only the first few axes gives a compact summary of the data, a way to plot high-dimensional data in 2-D, and a way to remove noise.

In this project you build PCA step by step, first with the covariance matrix and eigh (symmetric matrices), then with the SVD (SVD), and you check that the two agree. Then you project data, measure how much variance each component explains, measure reconstruction error (low-rank approximation), and compare your result with scikit-learn. Everything runs on a small synthetic data set with a fixed seed, so the numbers are reproducible.

PCA from scratch

Center Subtract each column's mean.
Decompose Eigenvectors of the covariance, or the SVD.
Rank Sort by variance, biggest first.
Project Keep the first k coordinates.
1 / 2

The data

We create 300 samples with 5 features. Behind them hide only two real factors (think "size" and "intensity"), mixed into five measured columns with some noise. A good PCA should discover that two components explain nearly everything. Rows are samples and columns are features, the usual shape for a data matrix X with n rows and d columns.

The function below is reused in every example on this page so each one is self-contained.

Python Create the data and look at it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The correlation matrix has many large entries, both positive and negative: the five columns carry much redundant information. That redundancy is what PCA removes.

Step 1: Center the data

PCA looks at the spread of the data around its mean, so the first step is to subtract each column's mean. Without it, the first "component" would simply point at the average of the data. If your columns are in very different units (centimetres versus kilograms), you would also divide by each column's standard deviation, which turns the covariance matrix into the correlation matrix. Here the columns are on comparable scales, so we only center.

Step 2: Covariance matrix and eigen-decomposition

For centered data Xc with n rows, the covariance matrix is

C = Xc^T Xc / (n - 1)

It is a symmetric d x d matrix: the diagonal holds each feature's variance and the off-diagonal entries show how pairs of features vary together. By the spectral theorem, C = Q D Q^T. The columns of Q are the principal axes (also called principal components or loadings), and the eigenvalues are the variances along those axes. np.linalg.eigh returns eigenvalues in ascending order, so reverse them to get the biggest first.

Python PCA through the covariance matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two eigenvalues dwarf the other three, and together they hold the large majority of the variance: exactly what the construction promised. The remaining three eigenvalues are near the noise level (the noise standard deviation was 0.3, so each noise direction has a variance near 0.09).

Step 3: The SVD route

Forming Xc^T Xc is the unstable move you met in numerical pitfalls: it squares the condition number. The SVD of the centered data matrix itself gives the same answer without ever forming the covariance. If Xc = U S Vt, then

C = Xc^T Xc / (n - 1) = V (S^2 / (n - 1)) V^T

So the rows of Vt are the principal axes, and the variances are S**2 / (n - 1). This is the version most libraries use.

Python The SVD gives the same axes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overlaps are all 1 (up to rounding), so the SVD and the covariance route find the same axes. Only the signs can differ, because an eigenvector v and -v are equally valid. Never compare principal axes between programs without allowing for that.

Step 4: Project the data

Projecting onto the first k axes means taking dot products of each (centered) sample with those axes. With the SVD this is one matrix product:

scores = Xc @ Vt[:k].T (shape n x k)

The scores are the data's coordinates in the new system. They are uncorrelated by construction: the covariance matrix of the full set of scores is diagonal, with the variances on the diagonal. To go back to the original space, multiply by the axes again and add the mean: X_hat = scores @ Vt[:k] + mean. The difference between X and X_hat is the information thrown away, and its size follows the Eckart-Young formula from the low-rank lesson.

Python Scores, reconstruction and error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reconstructing from only two components gives a small relative error: the five columns were really two-dimensional, plus noise. By k = 5 the reconstruction is exact. The error from the shortcut formula (the "predicted" column) matches the measured error, which confirms that PCA is simply the truncated SVD of the centered data.

Step 5: Compare with scikit-learn

scikit-learn's PCA class does exactly the steps above: it centers the data and runs an SVD. Its attributes use these names: components_ (the axes, as rows, like Vt), explained_variance_ (the variances), explained_variance_ratio_, and transform (the scores). Components may differ from yours in sign, so compare absolute values or compare projections.

Python Our PCA versus sklearn.decomposition.PCA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Everything matches. The ratios agree because both divide by the total variance of the full data, even though only two components were kept; scikit-learn computes the total from the singular values of all components just as we did.

Step 6: See the result

PCA is also a way to look at data. Plotting the first two scores shows the main structure of 5-D data on a flat page, and a scree plot (variance per component) shows where the elbow is.

Python Plotting scores and the scree curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The scree plot has a sharp elbow after the second bar, which is how you would choose k = 2 for real data. The scatter plot spreads out much more along the horizontal axis than the vertical one, as the variances predict.

Things to remember when using PCA

  • Scale matters. PCA follows variance, so a feature measured in large units dominates. Standardize columns (divide by the standard deviation) when units differ.
  • It only finds linear structure. Curved or clustered patterns need other methods.
  • Components are directions, not features. Each is a weighted mix of the original columns, and its sign is arbitrary.
  • High variance is not always high importance. PCA is unsupervised: it does not know what you want to predict.
  • Fit on training data only in a machine-learning pipeline, then apply the same centering and axes to new data with transform.

Practice these decompositions further in Project: PageRank, which uses eigenvectors of a very different matrix.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why must you center the data before computing PCA?

    The covariance matrix describes variation around the mean. Skipping the subtraction mixes the location of the data into the result.
  2. 2. If Xc = U S Vt is the SVD of the centered data (n rows), what are the variances along the principal axes?

    Xc^T Xc / (n - 1) = V (S^2 / (n - 1)) V^T, so the eigenvalues of the covariance matrix are S**2 / (n - 1).
  3. 3. How do you compute the scores (the data in the first k principal coordinates)?

    Each row of Xc is dotted with the first k axes (rows of Vt), giving an n x k matrix.
  4. 4. Your PCA and scikit-learn's give principal axes with opposite signs. What is the likely explanation?

    v and -v span the same line, so either sign is a correct axis. Compare absolute values, projections or reconstructions.
  5. 5. Why is the SVD of the centered data preferred to forming the covariance matrix?

    Forming Xc^T Xc squares the condition number, losing precision. The SVD works on Xc directly.

Summary

  • PCA = center the data, then take the eigenvectors of the covariance matrix Xc^T Xc / (n - 1) or, better, the SVD of Xc; the axes are the rows of Vt.
  • Variances along the axes are S**2 / (n - 1); the explained variance ratio is each one divided by the total.
  • Scores are Xc @ Vt[:k].T; reconstruction is Z @ Vt[:k] + mean, and its error is set by the dropped singular values.
  • scikit-learn's PCA does the same computation; signs of axes may differ, so compare absolute values or reconstructions.
  • Standardize features of different scales, expect only linear structure, and choose k from the scree plot or a cumulative variance target.

Next, see a very different use of eigenvectors in Project: PageRank.