Skip to content
Tutorials Center
My learning

Topics

Lesson 70 of 111 ยท Unsupervised Learning

Other Dimensionality Reduction

Go beyond PCA: TruncatedSVD for sparse text, NMF for parts-based topics, kernel PCA, random projections, factor analysis and ICA, with runnable scikit-learn examples.

PCA is the default tool for shrinking data, but it makes assumptions: the data is dense, centered, linear, and you want directions of maximum variance. Change one of those and a different method fits better. Sparse text matrices cannot be centered without becoming dense. Non-negative data such as word counts is easier to explain with parts that only add. Circles need a curved view. Mixed audio needs independent sources, not just uncorrelated ones.

This lesson tours the other members of sklearn.decomposition, plus random projections, with one runnable example each and a rule for when to reach for it.

Pick the method by the problem

TruncatedSVD Sparse text, no centering.
NMF Non-negative data, additive parts.
Kernel PCA Curved structure.
Random projection Fast, no fitting to data.
1 / 2

TruncatedSVD and latent semantic analysis

PCA subtracts the column means before computing the SVD. For a sparse matrix that would fill every zero with a non-zero number and blow up memory. TruncatedSVD skips the centering and works directly on sparse input. Applied to a TF-IDF matrix of documents it is called latent semantic analysis (LSA): each component is a "topic" mixing words that tend to appear together, and each document becomes a short dense vector.

NMF: parts that add up

Non-negative Matrix Factorization approximates a non-negative matrix V as W @ H, where both W and H contain only non-negative numbers. There are no cancellations, so each component is a pure additive "part": a topic is a set of words with positive weights, and a document is a positive blend of topics. That makes NMF topics usually easier to read than SVD components, which contain negative weights.

Python Topics from a tiny corpus: LSA and NMF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

NMF assigns each document mostly to one topic (football, computers or pets), and the top words per component read as clear themes. Component order is arbitrary. NMF is also sensitive to its starting point: it finds a local optimum, and with other random seeds two topics can merge or the numbering can change, so for real work try several seeds and compare the reconstruction error. (We start from a random initialization here because the smarter nndsvd starts hit a limitation in the in-browser build.)

Kernel PCA

Plain PCA is linear. KernelPCA applies the kernel trick first, so it effectively does PCA in a richer feature space without computing that space. With an RBF kernel, data that is not linearly flat, like two concentric circles, can be unfolded so that a straight line separates the rings. The cost is a full n x n kernel matrix, so it suits only a few thousand samples, and it has a gamma to tune.

Random projections

The Johnson-Lindenstrauss lemma says that you can map n points from a huge space into a much smaller one with a random matrix, and pairwise distances stay almost the same. GaussianRandomProjection and SparseRandomProjection need no fitting to the data's structure beyond its shape, so they are extremely fast and work on big sparse data. The result is noisier than PCA at the same size, but the trade is often worth it.

Python Kernel PCA on circles and random projections

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Linear PCA cannot pull the rings apart (their means coincide), so its first component scores about 0.5, which is chance. After the RBF map, the first component alone separates the rings perfectly. The right gamma matters: when we tried a much larger value the separation became poor, so tune it. For random projections, more dimensions keep distances more faithfully: the correlation with the true pairwise distances is about 0.61 with 8 random axes, 0.86 with 32 and 0.91 with 64.

Factor analysis

FactorAnalysis looks like PCA but comes from a different model: each observed feature is a weighted mix of a few hidden factors plus its own noise term, and different features may have different noise levels. PCA treats all variance as signal; factor analysis separates shared variance from feature-specific noise. That makes it better for survey-like data, where each question has its own measurement error, and it exposes the noise levels in noise_variance_.

ICA: separating independent sources

PCA finds uncorrelated directions. Independent Component Analysis (FastICA) goes further and looks for components that are statistically independent, which lets it undo a mixing of non-Gaussian sources. The classic example is the cocktail party: two microphones each record a blend of two speakers; ICA recovers the original voices up to order, sign and scale. It does not rank components by variance.

Python ICA unmixes two signals; factor analysis finds noise levels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

ICA recovers both waveforms almost perfectly (matches near 1), while PCA's components stay mixtures, because uncorrelated is not the same as independent. Factor analysis estimates a separate noise variance for every feature, very close to the values we planted (0.04, 0.25 and 0.64 for the clean, medium and noisy feature of each factor). It needs several features per factor to tell shared signal from private noise.

Factor loadings and rotation

A factor model writes each standardized feature as a weighted sum of a few hidden factors plus noise. The weights are the factor loadings: a loading of 0.9 says the feature is strongly tied to that factor. Reading the loading table is the whole point, because it tells you what each factor means (which features load on it). Two other numbers come with it. The communality of a feature is the share of its variance explained by the common factors (the sum of its squared loadings), and its uniqueness is what is left over, its private noise, which is exactly noise_variance_.

There is a catch. The mathematics of a factor model does not fix the orientation of the factors: any rotation of them fits the data equally well. An unrotated solution therefore tends to mix all features into every factor, which is hard to interpret. A rotation such as varimax turns the axes so that each feature loads strongly on one factor and weakly on the others. In scikit-learn, pass rotation="varimax".

The example creates six survey questions driven by two hidden traits (verbal and numeric ability) and recovers them.

Python Factor loadings, rotation and communality

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without rotation, every question loads on both factors (for example vocabulary at -0.66 and 0.68), so neither factor can be named. After varimax the picture is clean: vocabulary, reading and writing load above 0.79 on one factor, arithmetic, algebra and geometry load above 0.80 on the other, and all the cross-loadings are small. You can name the factors "verbal" and "numeric". The signs of loadings are arbitrary (the numeric factor came out negative), so interpret the size and the pattern, never the sign alone. Writing and geometry have the lowest communality (0.68 and 0.71), matching the larger noise we generated for them. Note the contrast with PCA, which explains 87% of the variance with two components because it counts everything as signal, while factor analysis reports that about 10% to 32% of each question is private noise.

Choosing quickly

Situation Try
Dense numeric data, general shrinking PCA
Sparse TF-IDF or counts, huge matrices TruncatedSVD
Non-negative data, want readable parts NMF
Curved structure, a few thousand rows KernelPCA
Enormous data, speed matters most Random projection
Per-feature noise, latent traits FactorAnalysis
Blind source separation FastICA

For plots of high-dimensional data see t-SNE and Manifold Learning. When you only need to select columns instead of building new ones, see Feature Selection.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use TruncatedSVD instead of PCA on a large sparse TF-IDF matrix?

    PCA centers the data, which turns a sparse matrix into a dense one. TruncatedSVD skips centering and keeps sparse input sparse.
  2. 2. What makes NMF components easy to interpret?

    With non-negative factors each item is an additive blend of parts, such as a document made of positive topic weights.
  3. 3. What does ICA assume that PCA does not?

    PCA only needs uncorrelated directions ordered by variance. ICA exploits independence of non-Gaussian sources to separate mixed signals.
  4. 4. When is a random projection a good choice?

    Random projections approximately preserve pairwise distances, cost almost nothing to fit, and scale to huge sparse data, but their components are not interpretable.

Summary

  • TruncatedSVD is PCA without centering; on TF-IDF it is called LSA.
  • NMF factors non-negative data into additive parts, giving readable topics.
  • KernelPCA handles curved structure at the price of an n x n kernel matrix.
  • Random projections preserve distances approximately and are very fast.
  • FactorAnalysis models hidden factors plus per-feature noise.
  • FastICA separates independent non-Gaussian sources, which PCA cannot.

Next, using these ideas to find the points that do not belong: Anomaly and Outlier Detection.