Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 76 ยท Cleaning and Preparing Data

Feature Selection: Filter, Wrapper and Embedded Methods

Choose the features that matter: filter methods (variance, correlation, mutual information), wrapper methods (RFE, forward selection) and embedded methods (Lasso, trees), without leakage.

More columns are not always better. Extra features slow training, make models harder to explain, invite overfitting, and cost money to collect and store. Feature selection means keeping the columns that help the model and dropping those that only add noise or repeat each other. It is different from feature engineering, which creates columns, and from dimensionality reduction, which combines columns into new ones such as principal components. Selection keeps a subset of the original, still readable columns.

There are three families of methods, which differ in how closely they involve the model: filter, wrapper and embedded. This lesson shows each one on the same data set, compares them, and explains the one mistake that spoils many projects: selecting features on the whole data set before cross-validation.

The data

We use the wine data set that ships with scikit-learn: 178 wines, 13 chemical measurements, three grape varieties as the target. To make the problem more realistic we add 10 pure-noise columns and 3 near-copies of existing columns, so a good method should ignore the noise and recognize the duplicates.

Python Wine data plus noise and duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Filter methods: score each feature on its own

A filter ranks features with a statistic that does not involve any model, then keeps the best. Filters are fast and scale to thousands of columns. Their weakness: they look at one feature at a time, so they can keep two redundant columns and miss two that only help together.

Common filters:

  • Variance threshold. A column that is (almost) constant carries no information. Remove columns with variance below a threshold. Compare variances only for features on the same scale, because variance depends on units.
  • Correlation with the target for numeric targets, or the ANOVA F-statistic (f_classif) for a categorical target: how different are the class means?
  • Chi-squared (chi2) for non-negative or count-like features against a categorical target.
  • Mutual information (mutual_info_classif) measures any dependence, including non-linear.
  • Correlation between features. If two columns have a correlation above, say, 0.9, keep one.
  • p-values from a statistical test can rank features, but with many features some will look significant by chance (see Hypothesis Testing); treat them as a ranking, not as proof.
Python Three filters on the wine data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The noise columns score far below the real ones, as hoped. Notice that the F-score also keeps flavanoids and its copy: a filter does not know they say the same thing, which is why the redundancy check is a separate step.

Wrapper methods: let a model judge subsets

A wrapper treats feature selection as a search. It trains the real model on a candidate subset, scores it (with cross-validation), and uses that score to choose the next subset. Because the model itself judges, wrappers account for interactions and redundancy, and the chosen features suit that specific model. The price is time: a model is trained many times.

  • Forward selection starts empty and adds the feature that improves the score most, until it stops helping.
  • Backward elimination starts with all features and removes the least useful one at a time.
  • Recursive feature elimination (RFE) repeatedly fits a model that gives coefficients or importances, removes the weakest feature(s) and refits.
Python RFE and forward selection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The selected columns are mostly real chemistry measurements (RFE may pick a copy instead of the original, which is equally good). Using scoring="neg_log_loss" for the forward search matters: with plain accuracy many subsets tie at the same value, and the search can then add a pure-noise column for no reason. A smoother score separates them. A model with a handful of columns scores about as well as one with all 26 (the accuracy here is slightly optimistic because the selection used the same rows; the leakage section below shows how to avoid that). Wrappers can be slow: forward selection with cv=5 fits many models per step, so on wide data use filters first to cut the candidate list.

Embedded methods: selection inside training

Embedded methods select features as a by-product of fitting the model.

  • L1 regularization (Lasso, L1 logistic regression) adds a penalty that pushes the coefficients of useless features to exactly zero. The features with non-zero coefficients are the selected ones. A stronger penalty (smaller C in logistic regression, larger alpha in Lasso) keeps fewer. In scikit-learn, LogisticRegression(l1_ratio=1, solver="saga") is the L1 version.
  • Tree importances. Random forests and gradient boosting report how much each feature reduced impurity. SelectFromModel keeps features above a threshold.
  • Permutation importance (see Model Interpretation) shuffles one column and measures how much the score drops; it works with any model and avoids a known bias of impurity importances toward high-cardinality columns.
Python L1 and tree-based selection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Do the selection inside cross-validation

A tempting shortcut: select the best features using all the data, then cross-validate a model on those features. This leaks information: the selection already peeked at the validation rows, so the score is too optimistic, sometimes wildly so when there are many features and few rows. The correct approach puts selection inside the pipeline, so it is refit on each training fold only (see Pipelines and Tuning).

Python Leaky versus honest selection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data is pure noise, so the true accuracy is about 0.5 (a coin flip). The leaky version reports a great score because the selection step found features that happen to match these particular labels. Only the honest pipeline tells the truth.

Supervised dimensionality reduction: LDA

Linear discriminant analysis (LDA) is a bridge between selection and dimensionality reduction. Like PCA it creates new columns as weighted sums of the originals, but instead of maximizing variance it maximizes the separation between classes. With k classes it can produce at most k - 1 components, and it needs labels. It is useful as a preprocessing step before a simple classifier, but the new columns are harder to read than selected original features.

Choosing a method

Situation Start with
Thousands of columns, need a quick cut Variance threshold, then a filter (F-score or mutual information)
Many correlated columns Correlation filter, or L1 regularization
Small data, expensive model is fine Wrapper (RFE or forward selection)
You already use a linear or tree model Embedded (L1, tree importances)
You must explain the model to others Selection (keeps original columns), not PCA or LDA

Whatever you choose, validate the final subset on data the selection never saw, and keep a note of why each column was dropped.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which statement describes a filter method?

    Filters rank features using a model-free statistic such as variance, correlation or mutual information. They are fast but ignore interactions and redundancy.
  2. 2. Why can selecting features on the full data set before cross-validation give a misleading score?

    Information from the validation rows leaks into the choice of features. Put the selector inside a pipeline so it is refit on each training fold only.
  3. 3. What does L1 regularization do for feature selection?

    The L1 penalty produces sparse coefficients, so features with a non-zero coefficient are the ones the model kept.
  4. 4. Two columns are almost exact copies of each other. Which tool is most likely to notice?

    A variance threshold and per-feature scores treat each column alone. Only a feature-to-feature correlation check reveals redundancy.
  5. 5. How is LDA different from feature selection?

    LDA is supervised dimensionality reduction. Selection keeps original columns; LDA replaces them with weighted combinations.

Summary

  • Feature selection keeps a useful subset of the original columns; it speeds up training, reduces overfitting and helps explanation.
  • Filters score features alone (variance, F-score, mutual information, correlation): fast, but blind to redundancy and interactions.
  • Wrappers (forward selection, backward elimination, RFE) let the model judge subsets: more accurate, slower.
  • Embedded methods (L1, tree importances) select while training.
  • Always do selection inside the cross-validation pipeline, or the score will be inflated.
  • LDA creates class-separating components instead of selecting columns.

Next: Missing Values: Strategies, or move on to Exploratory Data Analysis.