Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 111 ยท Support Vector Machines and Other Models

Discriminant Analysis

Learn linear and quadratic discriminant analysis in scikit-learn. Compare shared and per-class covariance, use LDA to reduce dimensions, and compare it with logistic regression.

Discriminant analysis is one of the oldest classification methods, and it is still a strong, fast baseline. The idea is to describe each class as a bell-shaped cloud of points (a Gaussian distribution), then classify a new point by asking which cloud it most likely came from. Because the Gaussian has a closed form, there is no iterative training: the model just computes class means, class priors and covariances from the training data.

Two flavors matter in practice. Linear discriminant analysis (LDA) assumes every class shares the same covariance (the same cloud shape), which produces straight decision boundaries. Quadratic discriminant analysis (QDA) lets each class have its own covariance, which produces curved boundaries. LDA has a bonus: it can also project your data onto a few directions that best separate the classes, so it doubles as a supervised dimensionality-reduction tool. This lesson builds on Naive Bayes, which also models each class with a distribution, and on Logistic Regression, its closest relative.

One cloud per class

Gaussian per class Each class gets a mean and a covariance.
Bayes' rule Pick the class with the highest posterior.
No iterations Fitting is just averages and covariances.
1 / 2

How it classifies

For each class k, discriminant analysis estimates three things from the training set: the prior (how common the class is), the mean vector (the center of the cloud) and the covariance matrix (its spread and tilt). To classify a point, it computes how plausible the point is under each class's Gaussian, weights that by the prior, and picks the winner. That is Bayes' rule with a Gaussian model of P(x | class).

The difference between the two methods is how the covariance is estimated:

  • LDA pools all classes into one covariance matrix. The quadratic terms cancel when you compare two classes, so the boundary is a straight line (a hyperplane in more dimensions).
  • QDA keeps a separate covariance matrix for each class. The quadratic terms stay, so the boundary is a curve (an ellipse, parabola or hyperbola).

The parameter count shows why this matters. With d features, LDA estimates one d x d covariance, but QDA estimates one per class. With many features and few samples, QDA's extra parameters are poorly estimated and the model overfits.

LDA versus QDA on clouds of different shape

Let us make two classes with deliberately different shapes: a round cloud and a stretched, tilted one. LDA's single shared covariance cannot describe both, while QDA can. The plot draws each model's decision regions.

Python Straight versus curved boundaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

QDA scores a few points higher (about 0.80 against 0.77) because the classes truly have different covariances. The classes overlap heavily, so neither model can reach a high score, but notice that the QDA region around the round cloud is curved while the LDA line is forced to be straight. On data where the classes really share a shape, the ranking usually flips or ties, and LDA's lower variance makes it the safer pick.

What the fitted model stores

After fit, the estimator exposes the learned Gaussian ingredients, so you can inspect them. This is a nice contrast with black-box models.

Python Look inside a fitted LDA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

priors_ are the class frequencies (equal on iris), means_ holds one row of feature averages per class, and coef_ has one row per class. Because the model is probabilistic, predict_proba works out of the box. Keep in mind that those probabilities rest on the Gaussian assumption; if the data is far from Gaussian, they can be overconfident (see Probability Calibration and Uncertainty).

Shrinkage for small or wide data

Estimating a covariance matrix is hard when you have many features and few samples. The raw estimate becomes unstable or even singular. LDA's solver="lsqr" or "eigen" accepts shrinkage, which blends the estimated covariance with a simple diagonal one. Use shrinkage="auto" to let the Ledoit-Wolf formula choose the blend. QDA offers a related knob, reg_param, that pulls each class covariance towards the identity.

LDA as dimensionality reduction

LDA has a second job. With K classes, there are at most K - 1 directions that carry class-separating information. LinearDiscriminantAnalysis(n_components=2).fit_transform(X, y) projects the data onto the best such directions: the ones that push class means apart while keeping each class tight.

Compare that with PCA from the unsupervised chapter (Principal Component Analysis). PCA looks only at X and keeps the directions of largest variance. LDA also looks at y and keeps the directions that separate the labels. That makes LDA supervised, and it also means you must fit it on training data only, never on the full dataset before splitting.

Python PCA versus LDA projections of the wine data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both plots use the same 13 wine measurements. PCA reports that its two directions hold about 36% and 19% of the total variance; LDA reports that its two directions carry about 69% and 31% of the class separation. The LDA picture usually shows the three wine types as tighter, better separated groups, because it was told where the labels are. explained_variance_ratio_ for LDA tells you how much of the between-class separation each direction carries, which is a different quantity from PCA's variance.

Using LDA to reduce dimensions before a model is a supervised step. Put it inside a pipeline so that cross-validation refits it on each training fold, otherwise label information from the test fold leaks into the features (see Pipelines).

LDA versus logistic regression

LDA and logistic regression give similar straight boundaries, so when do they differ? They estimate the same kind of model in different ways:

  • LDA is generative. It models how each class generates features, then applies Bayes' rule. If the Gaussian-with-shared-covariance assumption holds, it is the most efficient estimator and needs less data.
  • Logistic regression is discriminative. It models only P(class | x) directly and makes no assumption about how the features are distributed. It is more robust when the data is not Gaussian, for example with binary or heavily skewed features.

With small samples, many classes or near-Gaussian features, LDA often matches or beats logistic regression. With outliers or odd feature distributions, logistic regression tends to hold up better. Let us measure them on a small, wide dataset: 200 rows, 20 features, only 5 of them informative.

Python LDA, QDA and logistic regression compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 20 features and only 200 rows, plain QDA has to estimate two 20x20 covariance matrices from about 100 points each. It still lands near the others (about 0.75 against 0.73 for LDA and logistic regression), but the regularized version, reg_param=0.5, reaches about 0.82 by trading some flexibility for stability. Shrinkage barely changes LDA here because a single pooled covariance is already well estimated. The standard deviations show that differences of a few points between folds are noise, so always compare with cross-validation rather than trusting a rule of thumb.

When to use discriminant analysis

Reach for it when classes are roughly Gaussian, when you want a fast, deterministic baseline with no hyperparameters, when you need a supervised projection for plotting, or when you have many classes with limited data. Avoid it with many categorical or binary features, strongly non-Gaussian classes, or tiny classes (class covariance in QDA needs enough points per class to be stable).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the key assumption that makes LDA's decision boundary linear?

    With a shared covariance, the quadratic terms in the class comparison cancel, leaving a linear boundary. Independence given the class is the Naive Bayes assumption instead.
  2. 2. A dataset has few samples and many features. Which change is most likely to help QDA?

    QDA estimates one covariance per class, which is unstable with little data. reg_param pulls each covariance towards the identity and reduces variance.
  3. 3. How does LDA used as dimensionality reduction differ from PCA?

    PCA is unsupervised and looks only at X. LDA uses y and keeps the directions that best separate the class means.
  4. 4. A problem has 4 classes. What is the largest n_components LDA can use for projection?

    LDA yields at most min(n_classes - 1, n_features) components, so 3 for four classes (as long as there are at least 3 features).
  5. 5. Why should LDA used for feature reduction be placed inside a Pipeline when you cross-validate?

    LDA is supervised, so fitting it on all data first lets label information from the held-out fold shape the features. A pipeline refits it on each training fold only.

Summary

  • Discriminant analysis models each class as a Gaussian and classifies with Bayes' rule; fitting needs no iterations.
  • LDA shares one covariance across classes (linear boundaries); QDA uses one per class (curved boundaries, more parameters, more variance).
  • Use shrinkage (LDA) or reg_param (QDA) when data is small relative to the number of features.
  • LinearDiscriminantAnalysis(n_components=k) projects onto at most n_classes - 1 supervised directions; unlike PCA it uses the labels, so keep it inside a pipeline.
  • LDA is generative and efficient when its assumptions hold; logistic regression is discriminative and more robust when they do not.

Next up: Gaussian Processes, Neural Nets and Choosing a Model.