Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 111 ยท Feature Engineering and Pipelines

Scaling and Normalization

Learn when and how to scale features with StandardScaler, MinMaxScaler, RobustScaler, Normalizer and PowerTransformer, which models need it, and why you fit on train only.

Real datasets mix units. A house has 3 bedrooms and costs 450,000; a customer is 34 years old and earns 62,000. If a model compares these numbers directly, the larger scale wins by default, simply because its numbers are bigger. Scaling puts features on comparable footing so that no column dominates only because of its units.

Not every model cares. Some break without scaling, some are unaffected, and some just train faster. This lesson shows the main scalers in scikit-learn, demonstrates which algorithms need them, and explains the one rule that matters most: fit the scaler on the training data only. It leads into Encoding Categorical Features and, later, Pipelines.

Four common scalers

StandardScaler Mean 0, standard deviation 1.
MinMaxScaler Squeeze into 0 to 1.
RobustScaler Uses median and IQR, shrugs off outliers.
PowerTransformer Makes skewed data more bell-shaped.
1 / 2

The scalers

Every scaler follows the same API from The scikit-learn API: fit learns statistics from data, transform applies them, and fit_transform does both in one call. The learned numbers end in an underscore (mean_, scale_, data_min_, ...).

  • StandardScaler subtracts the mean and divides by the standard deviation: z = (x - mean) / std. The result has mean 0 and spread 1. It is the default choice for linear models, SVMs, k-NN and PCA.
  • MinMaxScaler maps the minimum to 0 and the maximum to 1: (x - min) / (max - min). It keeps the shape of the distribution and gives bounded values, handy for image pixels or neural-network inputs.
  • RobustScaler subtracts the median and divides by the interquartile range (the span of the middle 50% of values). A few extreme values barely change the median or the IQR, so the bulk of the data keeps a sensible scale.
  • MaxAbsScaler divides by the largest absolute value. It preserves zeros, which makes it suitable for sparse data.

The next example puts one extreme outlier into a tiny column and shows what each scaler does to the ordinary values.

Python Three scalers meet an outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With StandardScaler and MinMaxScaler the nine ordinary values are squeezed into a narrow band (about 0.08 wide on the 0-to-1 scale) because the outlier stretches the range. RobustScaler keeps them spread from -1 to about 0.8 and lets the outlier land far away, where it belongs. If your data has outliers that you cannot remove, use the robust scaler. If outliers are errors, fix or drop them first.

Which models need scaling?

The rule of thumb: scaling matters when the algorithm uses distances or gradient steps across features.

  • Distance-based models (k-NN, k-means, RBF-kernel SVM) add up differences between features. A feature in the thousands swamps one in single digits. See k-Nearest Neighbors.
  • Gradient-based models (linear and logistic regression with regularization, neural networks) converge faster and penalize coefficients fairly when features share a scale. Regularization such as Ridge and Lasso penalizes coefficient size, so unscaled features are penalized unevenly.
  • PCA finds directions of largest variance; without scaling it just finds the loudest unit.
  • Trees and forests split on thresholds one feature at a time. Multiplying a column by 1,000 changes the threshold values but not which splits are best, so their predictions do not change.
Python Who cares about scale? A cross-validated comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

k-NN and the SVM jump by 20 points or more once the features are scaled, logistic regression improves a little, and the tree and the forest score exactly the same. The scaler sits inside a pipeline here for a reason, covered next.

Fit on the training data only

The scaler's mean and standard deviation are learned from data. If you compute them from the whole dataset, then split, the training set has absorbed a bit of information about the test rows: a small leak. The correct order is:

  1. Split into train and test.
  2. scaler.fit(X_train): learn statistics from the training rows.
  3. scaler.transform(X_train) and scaler.transform(X_test): apply the same statistics to both.

The test data will therefore not come out with mean exactly 0 and standard deviation exactly 1, and that is correct: at prediction time you have no access to the new batch's statistics, so a new customer is scaled using the numbers learned in training.

Python Fit on train only, transform both

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The drift of +5 shows up as test means around 0.4 to 0.5 standard deviations above zero when scaled properly. Re-fitting on the test batch would erase that signal and make the model blind to the shift. In practice, you rarely manage this by hand: put the scaler in a Pipeline and the right thing happens automatically in every cross-validation fold.

Normalizer: scaling rows, not columns

Normalizer is different: it rescales each row (sample) to unit length. Two documents with the same word proportions but different lengths, or two embedding vectors that differ only in magnitude, become identical. It is stateless (nothing to learn), and it is common in text and similarity work. Do not confuse it with the casual use of "normalization" for min-max scaling.

PowerTransformer for skewed features

Incomes, prices and counts often have a long right tail. A power transform applies a monotonic function chosen to make the distribution closer to a bell curve, then standardizes it. PowerTransformer(method="yeo-johnson") works for any real values; method="box-cox" requires strictly positive data. A plain np.log is the simple alternative for positive data.

Python Taming a skewed feature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Skewness drops from a heavy positive value to about zero. This helps linear models and anything that assumes roughly symmetric inputs; trees do not need it.

Practical advice

  • Default to StandardScaler for linear models, SVMs, k-NN and PCA; use RobustScaler when outliers are real and stubborn.
  • Scale only numeric columns. One-hot encoded 0/1 columns usually do not need it. The ColumnTransformer lesson shows how to apply different steps to different columns.
  • Keep the fitted scaler together with the model. A model saved without its scaler gives nonsense on new data.
  • Interpret coefficients carefully: after standard scaling, a coefficient measures the effect of a one-standard-deviation change, which makes features comparable but changes the units.
  • Sparse data: avoid StandardScaler(with_mean=True), which would destroy sparsity; use MaxAbsScaler or with_mean=False.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which model's predictions are unaffected by multiplying one feature by 1,000?

    Trees choose split thresholds feature by feature, so a monotonic rescaling of a column changes the threshold values but not the resulting partitions. The other models depend on distances or penalized coefficients.
  2. 2. A column contains a few extreme outliers you cannot remove. Which scaler keeps the bulk of the values well spread?

    RobustScaler uses the median and interquartile range, which hardly move when a few values are extreme. Min-max and standard scaling are both stretched by the outliers.
  3. 3. You standardize the full dataset, then split it into train and test. What is wrong?

    The mean and standard deviation are learned statistics. Computing them on all rows lets the test data shape the transformation. Fit on the training data and reuse those numbers for the test data.
  4. 4. What does Normalizer do?

    Normalizer works row by row, rescaling every sample vector to norm 1. Column-wise scaling is done by the other scalers.

Summary

  • Scaling puts features on comparable scales so units do not decide which feature dominates.
  • StandardScaler (mean 0, std 1) is the default; MinMaxScaler bounds values; RobustScaler resists outliers; Normalizer scales rows; PowerTransformer reduces skew.
  • Distance-based, gradient-based and variance-based methods (k-NN, SVM, linear models, PCA) need scaling; trees and forests do not.
  • Always fit the scaler on the training data and reuse its statistics for validation, test and new data; a pipeline automates this.
  • Test data scaled with training statistics will not have mean exactly 0, and that is how it should be.

Next: Encoding Categorical Features.