Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Pipelines and Feature Engineering

Feature Engineering and Feature Selection

Improve scikit-learn models with encoders, scalers, polynomial features, binning and imputation, then trim features with SelectKBest, RFE, L1 and permutation-based selection.

Better inputs usually beat fancier models. Feature engineering is the craft of turning raw columns into inputs a model can use well: encoding categories, scaling numbers, filling gaps, creating interactions. Feature selection is the opposite move: removing columns that add noise, cost or complexity without adding signal.

All of the tools below are transformers, so they slot into a pipeline and are refitted inside cross-validation.

Scaling numbers

Scaler What it does Use when
StandardScaler Mean 0, standard deviation 1 Default for linear models, SVMs, k-NN, PCA
MinMaxScaler Squeezes into a range, usually 0-1 Bounded inputs, neural nets
RobustScaler Uses median and IQR Outliers present
PowerTransformer Makes skewed data more Gaussian Heavy skew
QuantileTransformer Maps to uniform or normal distribution Strange shapes
Normalizer Scales each row to unit length Text, cosine similarity

Tree-based models (decision trees, random forests, gradient boosting) do not need scaling.

Python Scalers compared on a skewed, outlier-heavy column

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With outliers, MinMaxScaler crams most data into a tiny range near 0, while RobustScaler and PowerTransformer keep the bulk well spread.

Encoding categories

  • OneHotEncoder: one 0/1 column per category; the standard for nominal data. Use handle_unknown="ignore" and, for rare levels, min_frequency= or max_categories=.
  • OrdinalEncoder: integers; use it for naturally ordered categories (categories=[["low","mid","high"]]), or with tree models.
  • TargetEncoder (scikit-learn 1.3+): replaces a category by a cross-fitted average of the target; helpful with thousands of levels.
Python One-hot, ordinal and target encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Target encoding uses the label, so it must be fitted only on training data (inside a pipeline) and uses internal cross-fitting to limit leakage.

Missing values

SimpleImputer fills with a mean, median, most frequent value or constant. KNNImputer uses similar rows. IterativeImputer (experimental) models each column from the others. Adding add_indicator=True appends a flag column "was missing", which can itself be predictive. Some models (HistGradientBoosting*) handle NaN natively.

Creating features

Python Polynomial features, binning and splines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A straight line cannot follow a wave, but the same linear model on expanded inputs can. Polynomial features also create interactions (x1*x2) when you have several columns; interaction_only=True keeps products and skips powers. Splines are a smoother, better-behaved alternative to high-degree polynomials.

Other useful ideas: ratios and differences, date parts (month, weekday, is_weekend), cyclic encoding of hours with sine and cosine, log transforms of money, text length, counts per group. See custom transformers for packaging them.

Feature selection

Why remove features? Fewer columns mean faster training, easier explanation and less overfitting, especially when features outnumber rows.

Three families of selection

Filter Score each feature alone, keep the best.
Wrapper Retrain with subsets (RFE).
Embedded The model selects (L1, trees).

Always select inside cross-validation, never on all the data first.

1 / 2
Python Filter, wrapper and embedded selection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first 8 columns (5 informative + 3 redundant) are the real signal. The filters and the forest pick mostly indexes below 8. The linear wrapper (RFE) and L1 may also pick noise columns when the signal is spread over correlated columns, a reminder to validate any selection with cross-validation instead of trusting it blindly.

Selection inside a pipeline

Python Does selection help? Cross-validated comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When columns far outnumber rows, keeping a modest k usually beats keeping everything.

Tips

  • Remove obviously useless columns first: constants, IDs, near-duplicates (VarianceThreshold, correlation checks).
  • Univariate filters ignore interactions; a feature useless alone can be vital with another.
  • Model-based importances can be biased toward high-cardinality features; use permutation importance on held-out data to double-check.
  • Selection reduces input cost but is not the same as explaining the model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which scaler is least affected by outliers?

    RobustScaler uses the median and interquartile range, which extreme values barely move.
  2. 2. A naturally ordered column (low, mid, high) is best encoded with

    OrdinalEncoder preserves the order when you pass the categories list explicitly.
  3. 3. Why must feature selection happen inside cross-validation?

    This is a form of leakage. In a pipeline, selection is redone on each training fold.
  4. 4. Which selector retrains a model repeatedly, dropping the weakest features each time?

    Recursive feature elimination (RFE) is a wrapper method that removes the least important features step by step.

Summary

  • Scale for distance- and gradient-based models, not for trees; choose robust scalers with outliers.
  • Encode categories with one-hot, ordinal or target encoding; impute missing values and consider indicator flags.
  • Polynomial, spline and bin transformers let linear models fit curves.
  • Selection comes in filter, wrapper and embedded flavors; always do it inside the pipeline.

Next: Text Feature Extraction