Skip to content
Tutorials Center
My learning

Topics

Lesson 68 of 111 ยท Unsupervised Learning

Principal Component Analysis

Learn principal component analysis: find directions of maximum variance, read explained_variance_ratio_, choose components, whiten, reconstruct, and build PCA from SVD.

Real datasets often have many columns that carry overlapping information. Height and weight move together; pixels next to each other have similar brightness. Principal Component Analysis (PCA) finds a new set of axes for your data, ordered so that the first axis captures as much variation as possible, the second captures as much of what is left as possible, and so on. Keep only the first few axes and you have a smaller dataset that still holds most of the information.

PCA is used to visualize data in two dimensions, to remove noise, to speed up other models, and to decorrelate features. It is unsupervised: it never looks at a target. It builds on the ideas in Scaling and Normalization and on the clustering lessons that came before, such as k-Means Clustering.

What PCA does

Finds directions Axes along which the data varies most.
Ranks them First component has the most variance.
Keeps a few Drop the weak directions, keep the signal.

PCA rotates your data, then lets you cut the least useful axes.

1 / 2

Variance and directions

Imagine a cloud of points stretched like a cigar and tilted. The cigar's long axis is the direction of greatest variance: if you had to describe each point with a single number, the position along that axis would tell you the most. That axis is the first principal component. The second is perpendicular to the first and captures the most remaining variance. In d dimensions you get d perpendicular components, and together they are just a rotation of your original axes.

Each component is a weighted mix of the original features. The weights are stored in pca.components_, one row per component, and the amount of variance each one captures is in pca.explained_variance_ (the ratio form is explained_variance_ratio_).

Python Principal components of a tilted cloud

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first ratio is close to 0.95: nearly all the variation lies along one tilted line, so a single number per point loses very little. The red arrows are the rows of components_, scaled by the spread along each.

Component signs are arbitrary. A component and its negative describe the same axis, so do not be surprised if a different library version flips a sign.

Always scale first

PCA maximizes variance, and variance depends on units. A column measured in grams has a variance a million times larger than the same quantity in kilograms. Without scaling, the biggest-number column simply becomes the first component. Put a StandardScaler before PCA, ideally in a pipeline, unless all columns already share a unit and the raw scale is meaningful (pixel intensities, for example).

Choosing how many components

Two common rules. Look at the cumulative explained variance and keep enough components to reach a target such as 90% or 95%. Or look at the scree plot (variance per component) and cut where the curve flattens into an "elbow". scikit-learn lets you pass a float to n_components to apply the first rule directly: PCA(n_components=0.95) keeps just enough components to explain 95% of the variance.

The digits dataset has 64 pixel columns per 8x8 image, which makes it a good testbed.

Python How many components do the digits need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Roughly 29 components, fewer than half of the 64 pixels, hold 95% of the information. Neighbouring pixels are strongly correlated, and some corner pixels are always blank, so most of the 64 dimensions are redundant.

Reconstruction and what you lose

After transform, inverse_transform maps the reduced data back to the original space. The result is not identical to the input: the dropped components are gone, so you get the closest approximation that the kept axes can express. The average squared gap between original and reconstruction is the reconstruction error, and it equals the variance you threw away. This is why PCA doubles as a denoiser: small, noisy directions are discarded.

Whitening

After projecting, the components have very different spreads (the first much larger than the last). PCA(whiten=True) divides each component by its standard deviation, so every output column has variance 1 and the columns are uncorrelated. Some algorithms, such as k-means or SVMs with an RBF kernel, behave better on whitened data. The price is that whitening amplifies tiny, noisy components, so use it together with a limit on n_components.

PCA from SVD in NumPy

Under the hood PCA is a singular value decomposition (SVD) of the centered data matrix. If Xc = X - X.mean(axis=0) and Xc = U @ diag(S) @ Vt, then the rows of Vt are the principal components, the variance of component i is S[i]**2 / (n - 1), and the projected data is Xc @ Vt.T, which equals U * S.

Python PCA by hand matches scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two numbers in each reconstruction line agree to within a hair (the tiny gap is n versus n - 1 in the variance formula): the error you pay is the variance you dropped.

Practical advice

  • Fit PCA on training data only, inside a pipeline, so the test set never influences the axes.
  • Components are mixes of all features, so they are harder to explain than the original columns. Inspect components_ to see which features load heavily on each one.
  • For very large data, PCA(svd_solver="randomized") is much faster, and IncrementalPCA works in chunks. For sparse matrices use TruncatedSVD, covered in Other Dimensionality Reduction.
  • High variance is not the same as usefulness for prediction. A low-variance direction can still separate your classes, so check the downstream score.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the first principal component represent?

    PCA is unsupervised. The first component is the direction along which the centered data is most spread out.
  2. 2. Why should you usually standardize features before PCA?

    A column in large units has a huge variance and would capture the first component by itself, whatever its real importance.
  3. 3. What does PCA(n_components=0.95) do?

    A float between 0 and 1 is treated as a target for the cumulative explained variance ratio.
  4. 4. Where must PCA be fitted to avoid data leakage?

    The axes are learned from data, so fitting on the test set leaks information. Use a pipeline so cross-validation refits PCA on each training fold.

Summary

  • PCA rotates the data onto perpendicular axes ordered by variance; keeping the first few gives a compact summary.
  • explained_variance_ratio_ and its cumulative sum guide how many components to keep; n_components=0.95 automates it.
  • Scale features first and fit PCA on training data only.
  • inverse_transform reconstructs an approximation; the error equals the discarded variance.
  • whiten=True gives unit-variance, uncorrelated outputs.
  • PCA is an SVD of the centered data matrix.

Next, curved structure that PCA cannot flatten: t-SNE and Manifold Learning.