Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Unsupervised Learning

Principal Component Analysis (PCA)

Reduce dimensions with scikit-learn PCA: explained variance, choosing components, scaling, inverse_transform for compression, visualization, TruncatedSVD and KernelPCA.

Real datasets often have dozens or thousands of columns, and many of them move together: height and weight, pixel next to pixel, word counts for related words. Principal component analysis (PCA) rewrites the data using a new set of axes, ordered by how much variation each one carries. Keeping only the first few gives you a smaller dataset that still holds most of the information.

People use PCA to visualize data in 2-D, to compress features, to remove noise, to speed up other models, and to fight collinearity.

The idea

PCA finds the direction in which the data varies the most; that is principal component 1. Then it finds the direction of greatest remaining variation that is perpendicular to the first, and so on. Each component is a weighted mix of the original features. Projecting the data onto the first k components gives k new columns.

What PCA does

New axes Directions of greatest variance.
Orthogonal Components are uncorrelated.
Keep k Drop the weak, noisy ones.

PCA is unsupervised: it never looks at the target.

1 / 2

A first PCA

Python PCA on the iris measurements

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

explained_variance_ratio_ is the share of the total variance each component keeps. components_ holds the weights: a large absolute weight means that feature contributes strongly to the component. Four measurements shrink to two columns while keeping most of the variance.

How many components?

Look at the cumulative explained variance and pick the smallest k that reaches a target such as 95%. You can also pass a float: PCA(n_components=0.95) keeps just enough components for 95%.

Python Cumulative explained variance on digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 30 of the 64 pixel columns carry 95% of the variation; the rest is mostly noise and redundancy.

Compression and denoising

inverse_transform maps the reduced data back to the original space. The result is an approximation: what the components can reconstruct. The error shows what was thrown away.

Python Reconstruction error versus components

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real structure lives in a few strong components, while random noise spreads thinly over all of them. Keeping only the strong ones throws away much of the noise.

Visualizing high-dimensional data

Projecting onto two components lets you draw data with 64 or 4,000 features. Classes that separate in the plot are probably learnable; a blob suggests the structure is not linear.

Python Digits in two dimensions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

PCA inside a pipeline

Treat PCA as a preprocessing step, and fit it only on training data like any other transformer. You can tune n_components with a search.

Python PCA + classifier, tuned

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • TruncatedSVD: like PCA but works directly on sparse matrices without centering; the standard tool for text (latent semantic analysis).
  • IncrementalPCA: fits in batches when data does not fit in memory.
  • KernelPCA: nonlinear structure by applying a kernel first.
  • PCA(svd_solver="randomized"): faster approximate solver for big matrices.

Limits

  • Components are linear mixtures; curved structure needs manifold methods.
  • Components can be hard to interpret; high variance is not the same as high predictive value.
  • It is sensitive to scale and to outliers, so standardize and inspect first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the first principal component represent?

    PCA is unsupervised: components are ordered by how much variance they capture, not by relation to a target.
  2. 2. What does PCA(n_components=0.95) do?

    A float between 0 and 1 is treated as the target share of explained variance.
  3. 3. Why standardize features before PCA?

    PCA maximizes variance, which depends on units. Standardizing gives each feature a fair say.
  4. 4. Which tool fits PCA-like reduction directly on a sparse text matrix?

    TruncatedSVD does not center the data, so sparsity is preserved.

Summary

  • PCA finds orthogonal directions of maximum variance and lets you keep only the strongest.
  • Standardize first, then choose components by cumulative explained variance.
  • inverse_transform reconstructs data for compression and denoising.
  • Use it for 2-D plots, speed, and collinearity, and fit it only on training data in a pipeline.
  • Use TruncatedSVD for sparse data and manifold methods for nonlinear structure.

Next: Anomaly Detection and Manifold Learning