Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 111 ยท Feature Engineering and Pipelines

Feature Selection

Choose which features to keep: VarianceThreshold, SelectKBest, RFE, SelectFromModel and L1 selection, and how to select inside cross-validation without leakage.

More columns are not always better. Some features carry no information, some repeat others, and some only add noise that a flexible model will happily memorize. Feature selection means keeping a useful subset of the columns and dropping the rest. A smaller model is faster, easier to explain and sometimes more accurate, especially when you have many columns and few rows.

scikit-learn groups the methods into three families. Filters score each feature on its own, before any model is trained. Wrappers train a model repeatedly and drop features step by step. Embedded methods let the model's own training pick features. This lesson tries one or two of each, and ends with the most important rule: selection is part of training, so it must happen inside cross-validation.

Three families of selection

Filters Score every column alone. Fast and model-free.
Wrappers Fit a model again and again, dropping the weakest.
Embedded The model picks features while it trains.

Cheaper methods first; wrappers cost many model fits.

1 / 2

Filter: drop constant columns

The simplest filter is VarianceThreshold. A column that is constant, or nearly so, cannot separate anything. The default threshold of 0.0 removes only columns whose value never changes. It looks at the features alone and never at the target, so it is safe to run on any data.

Python VarianceThreshold removes constant columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Remember that variance depends on the scale of the column. Compare variances only on features with similar scales (Scaling and Normalization).

Filter: SelectKBest

SelectKBest(score_func, k=...) scores each feature against the target and keeps the top k. The score function must fit the problem: f_classif (an ANOVA F-test) and mutual_info_classif for classification, f_regression and mutual_info_regression for regression. The F-tests only see linear relationships; mutual information also catches nonlinear ones but is noisier and slower.

Python SelectKBest with two scoring functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The F-test finds columns 1, 2 and 3, but misses column 0 and picks a noise column instead: column 0 is informative, yet the class averages in that column almost coincide, and the F-test only compares averages. Mutual information, which also sees other kinds of dependence, finds all four. A filter also ranks features one at a time, so it can miss a pair that is only useful together, and it can keep two nearly identical features because each one scores well alone.

Wrapper: recursive feature elimination

RFE (recursive feature elimination) fits a model that exposes feature weights or importances, removes the weakest feature (or a few), and repeats until the requested number remains. It is more thorough than a filter because the model judges each feature in the company of the others, but it needs many fits.

Python RFE with a logistic regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ranking_ attribute shows the order of elimination: all kept features have rank 1, the last one removed has rank 2, and so on. RFECV chooses the number of features itself with cross-validation.

Embedded: SelectFromModel and L1

SelectFromModel keeps the features whose weight or importance in a fitted model is above a threshold (the mean by default). It works with any estimator that has coef_ or feature_importances_.

An L1 penalty (Ridge, Lasso and Elastic Net) pushes the weights of unhelpful features to exactly zero, so a Lasso or an L1 logistic regression selects features as a side effect of training. Pair it with SelectFromModel to extract the survivors. Scale the features first, otherwise the penalty treats columns unequally.

Python L1 selection with SelectFromModel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Embedded selection depends on the model that does it. Lasso selects what a linear model can use; a forest may select a different set. If you want to feed the chosen features to a different model, test that combination.

More filter scores: Fisher score and dispersion ratio

Two older filter scores are quick to compute by hand and good for building intuition.

  • The Fisher score of a feature measures how well it separates the classes: the spread between the class means divided by the spread within the classes. For class c with n_c rows, mean m_c and variance s_cยฒ, and an overall mean m: F = ฮฃ n_c (m_c - m)ยฒ / ฮฃ n_c s_cยฒ. A large F means the classes sit far apart relative to their own scatter. (This is the same idea as the ANOVA F-test behind f_classif.)
  • The dispersion ratio compares the arithmetic mean of a feature with its geometric mean (for positive features). The ratio is 1 when all values are equal and grows with the spread, so a ratio close to 1 marks an uninformative, near-constant feature. It ignores the target, like VarianceThreshold.
Python Fisher score and dispersion ratio by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Features such as flavanoids and proline separate the three wine classes well (high Fisher score), while others such as ash barely do. The dispersion ratio looks at the feature alone and is only useful for spotting near-constant columns: alcohol has a ratio of 1.00, because its values are all of similar size, yet it still has a high Fisher score, since its small differences line up with the classes. A large ratio would also be possible for a feature that varies a lot but says nothing about the class. Filter scores are fast, but they look at one feature at a time and miss interactions and redundancy: two perfectly correlated features both get a high score.

Forward selection and backward elimination

Wrapper methods evaluate subsets of features by actually training a model. Two greedy strategies are the standard ones:

  • Forward selection starts with no features. At each step it adds the feature that improves the cross-validated score the most, and stops after a chosen number of features or when no feature helps.
  • Backward elimination starts with all features and removes, at each step, the feature whose removal hurts the least.

Both are available as SequentialFeatureSelector (direction="forward" or "backward"). They cost many model fits (forward selection of k features out of p fits about k ร— p models, times the number of cross-validation folds), so they suit modest feature counts. They are greedy, not exhaustive: the best pair of features need not contain the best single feature.

Python Forward selection and backward elimination with SequentialFeatureSelector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four features keep most of the accuracy of all fifteen (0.94 for forward selection and 0.92 for backward elimination, against 0.95), and the two directions pick different subsets: forward selection grows a set one feature at a time, backward elimination prunes one, and with correlated features the greedy paths diverge. When you report such a result, remember that selecting the features with the same data you validate on can be optimistic. Do the selection inside each cross-validation fold (leakage-safe selection).

Backward elimination with p-values

For linear regression, statisticians use a classic variant: fit the model with all features, look at the p-value of each coefficient, remove the feature with the largest p-value if it is above a threshold (often 0.05), refit and repeat until all remaining features are significant. It is fast and gives a readable final model, but it inherits the weaknesses of p-values (they depend on the assumptions listed in linear regression) and it is a selection step that must not touch the test set.

Python Backward elimination by p-value (statsmodels)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The procedure removes noise columns one at a time, always the least significant one, and stops with the three genuinely useful features. If a noise column happens to look significant by chance (about 1 in 20 at the 0.05 level), it will survive, which is another reason to confirm the result on held-out data.

When does selection help?

Selection helps most when you have many more features than rows, when many features are pure noise, when you need a model that people can read, or when prediction speed matters. It often makes little difference for tree ensembles and regularized linear models, which already ignore useless features reasonably well. It can even hurt when features are weak individually but strong together.

Treat the number of features to keep, such as k, as a hyperparameter and tune it with cross-validation (Hyperparameter Tuning).

Leakage-safe selection inside cross-validation

Feature selection looks at the target, so it is a learning step, just like fitting a model. If you select features on the whole dataset and then cross-validate, the selector has already seen the labels of the validation rows, and your score is too optimistic. The effect is strongest with many features and few samples.

The experiment below uses pure noise: 2,000 random features and random labels. No honest model can beat 50% accuracy. Yet selecting the 20 "best" features first looks like a success.

Python The classic selection leak

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first number is far above 0.5 even though there is nothing to learn. The second is honest. The fix costs one line: make the selector a step of a pipeline, so every training fold chooses its own features and the validation fold stays untouched. The general lesson is in Nested Cross-Validation and Data Leakage.

Any step that learns from the target (selectors, target encoders, supervised binning) belongs inside the pipeline. Steps that only look at the features, such as a fixed VarianceThreshold(0.0), are lower risk, but putting them in the pipeline costs nothing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which of these is a filter method?

    A filter scores features without training the final model. RFE is a wrapper, while Lasso and SelectFromModel are embedded methods because the model itself picks the features.
  2. 2. What does VarianceThreshold(threshold=0.0) remove?

    With a threshold of 0.0 it drops only columns whose variance is zero, meaning every row has the same value. It never looks at the target.
  3. 3. You select the top 20 features on the full dataset, then run cross-validation. Why is the score unreliable?

    Selection uses the target, so it must be refit inside each training fold. Doing it first leaks validation labels into the choice of features.
  4. 4. What does the ranking_ attribute of a fitted RFE show?

    Kept features have rank 1; the last feature removed gets rank 2, the one before it rank 3, and so on.
  5. 5. When is feature selection most likely to help?

    With many noisy features and few rows a flexible model can fit random patterns. Dropping useless columns reduces that risk.

Summary

  • Selection keeps a useful subset of features: filters (VarianceThreshold, SelectKBest), wrappers (RFE) and embedded methods (SelectFromModel, L1 penalties).
  • Match the score function to the task (f_classif, f_regression, mutual information) and scale features before L1 or variance-based selection.
  • Selection helps most with many noisy features and few rows; trees and regularized models often gain little.
  • Treat the number of features as a hyperparameter.
  • Selection learns from the target, so it belongs inside the pipeline, refit on every training fold.

Next: Pipelines.