Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 76 ยท Modelling

Dimensionality Reduction and Visualization of High-D Data

Reduce many columns to a few with PCA: explained variance, scree plots, loadings, PCA in a pipeline, and why t-SNE pictures need careful reading.

A table with 5 columns is easy to inspect; a table with 50 or 500 columns is not. You cannot draw 50 axes, many columns often repeat the same information (height, weight and shoe size mostly measure body size), and models with a huge number of inputs can overfit. Dimensionality reduction replaces many columns with a few new ones that keep as much of the useful structure as possible.

This lesson focuses on PCA (principal component analysis), the standard first tool, and ends with t-SNE, a popular method for drawing pictures of high-dimensional data that is easy to over-read. Reduction is an unsupervised technique, like clustering: it uses only the feature table, never a target column. You will use it for three jobs: seeing the data, compressing it, and understanding which columns move together.

The idea behind PCA

Imagine a cloud of points. PCA finds the direction along which the cloud is widest; that is the first principal component (PC1). The second component is the widest direction at a right angle to the first, and so on. Each component is a weighted sum of the original columns, and each new column (the score) says where a row lies along that direction. Keeping only the first few components gives a smaller table that retains most of the variation.

flowchart LR
    A["Table: n rows x p columns"] --> B["Standardize<br/>columns"]
    B --> C["PCA finds orthogonal<br/>directions of most variance"]
    C --> D["Keep first k components"]
    D --> E["New table: n rows x k scores"]

PCA rotates the data so the first axes carry the most variance; you keep the first k.

Two facts to remember. PCA is based on variance, so scale the columns first, otherwise the column with the largest units wins. And PCA is linear: it finds straight-line structure only.

Explained variance and the scree plot

Each component explains a share of the total variance, available as explained_variance_ratio_. The shares add up to 1, and they are sorted from largest to smallest. A scree plot shows them as bars; the cumulative sum tells you how many components cover, say, 90% of the variance.

The synthetic table below has six columns, but only three real sources of variation: body size drives height, weight and shoe size; training habits drive gym hours and protein intake; and one column is pure noise. A good PCA should find about three important components.

Python Explained variance, with and without scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With standardized columns, the first three components explain about 94% of the variance, and the next three together explain only 6%, which matches the three sources we built in. Without scaling, the first component claims 77% only because protein (in grams, with a very large spread) has a far bigger raw variance than the other columns, and the noise column and the shoe size almost vanish. The raw numbers measure units, not structure.

How many components should you keep? Common rules: enough to reach a target such as 90% of the variance; the "elbow" in the scree plot; or the number that gives the best score in a downstream model, checked with cross-validation. Scikit-learn also accepts PCA(n_components=0.9) to pick the number for a variance target.

Loadings: what a component means

The loadings (in pca.components_) are the weights that turn columns into a component. Large values, positive or negative, show which columns define it. Reading the loadings is how you give components names and how you find groups of related columns.

Python Reading loadings and plotting scores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

PC1 gives almost equal weight to height, weight and shoe size: it is a "body size" axis. PC2 is dominated by gym hours and protein: a "training habits" axis. PC3 is nearly all noise_a, so it carries no shared structure; it appears only because the noise column has the same variance as everything else after scaling. In the scatter plot the colour changes along PC2, as expected.

Two cautions when reading components:

  • The sign is arbitrary. A component and its mirror image are equally valid, so never say "positive PC1 means big" without checking the loadings of your own run.
  • Components are not features of importance for a target. PCA never saw a target. A low-variance component can still be the one that predicts your outcome; high variance only means high spread.

PCA needs numeric columns without missing values. Impute missing data and encode categories first, and fit the scaler and PCA on the training data only, so no information leaks from the test set.

PCA inside a pipeline

You can use PCA as a preprocessing step to compress many correlated inputs before a model. Put it in a Pipeline so that it is fitted only on training rows. The example uses scikit-learn's small built-in handwritten digits (8 x 8 pixel images, so 64 columns) and compares a classifier on all pixels against one on 20 components.

Python Compressing 64 pixel columns to 20 components

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line of output shows that 30 of the 64 columns are needed to keep 90% of the variance. Reducing the inputs to 20 components costs about two points of accuracy (0.972 down to 0.952) and 5 components lose a lot (0.815). This is the typical trade-off: compression helps speed and stability, but the number of components is a hyperparameter you tune, not a fact. For models that already handle many inputs well (trees, regularized regression), PCA often gives no gain and makes the inputs harder to explain.

t-SNE: pictures, with caution

PCA is linear and can miss curved structure. t-SNE (t-distributed stochastic neighbor embedding) places points in 2D so that neighbours in the original space stay neighbours on the page. It often produces striking pictures with visible clumps. It is also easy to misread, so treat it as a drawing tool, not as an analysis:

  • Distances between clumps mean little. Only local neighbourhoods are preserved; a big gap or a close pair of clusters on the map may not exist in the data.
  • Cluster size and density are not reliable. The method spreads dense regions out.
  • The result depends on settings. perplexity (roughly, how many neighbours each point considers) and the random seed change the picture, so try several and trust only features that appear in all.
  • There is no transform for new rows. You embed a fixed set of points; you cannot place tomorrow's data on the same map.
  • It is slow on big tables. It is common to reduce with PCA to 30-50 columns first.

The example draws the same 700 digits three ways: PCA, t-SNE with a low perplexity and t-SNE with a larger one. It takes a few seconds to run, because t-SNE is slow even for small tables.

Python PCA versus t-SNE on handwritten digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two PCA axes keep only about 28% of the variance, so digits overlap heavily. Both t-SNE maps separate the classes much better, but they are not identical: the layout, the spacing between groups and which digits sit next to each other differ. That is the instability to remember. If a conclusion depends on one particular t-SNE picture, it is not a conclusion yet.

Another popular method, UMAP, behaves similarly and is faster, but it is a separate package that is not available in the browser runner. The same cautions apply to it.

Choosing a method

Goal Reasonable choice
Compress correlated numeric columns, keep a linear model PCA
Understand which columns move together PCA loadings, or a correlation matrix
Draw a picture of high-dimensional clusters PCA first, then t-SNE or UMAP, with several settings
Keep original, interpretable columns Feature selection, not PCA

Remember what PCA costs: the new columns are blends, so "PC2" is harder to explain to a stakeholder than "hours in the gym". Use it when compression or visualization is worth that loss.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is it important to standardize columns before PCA?

    PCA maximizes variance, which depends on units. Without scaling, a column measured in large numbers dominates the first components regardless of how informative it is.
  2. 2. The cumulative explained variance of the first 3 components is 0.94. What does this mean?

    Explained variance describes how much of the spread of the original columns is retained. It says nothing about accuracy for a target.
  3. 3. What do loadings tell you?

    A loading is the weight of one original column in a component. Large positive or negative weights show which columns define it.
  4. 4. Which statement about a t-SNE plot is the safest?

    t-SNE preserves local neighbourhoods. Gaps, cluster sizes and global layout are not reliable and change with settings.
  5. 5. Where should the PCA step be fitted when you build a predictive model?

    Fitting on all data lets test-set information shape the components, a form of leakage. A pipeline fits each step on the training rows only.

Summary

  • Dimensionality reduction replaces many columns with a few that keep most of the structure; use it to see, compress and understand data.
  • PCA finds orthogonal directions of maximum variance; scale the columns first.
  • explained_variance_ratio_ and a scree plot help you choose how many components to keep.
  • Loadings show which columns define each component; the sign of a component is arbitrary.
  • Put PCA inside a pipeline so it is fitted on training data only, and tune the number of components like any hyperparameter.
  • t-SNE draws useful pictures but distances, sizes and layout are unreliable; try several settings and do not base conclusions on one map.

Next: Forecasting Basics, where rows have a time order and the future is the target.