Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 64 ยท Transforming Data

Normalizing and Scaling Columns

Scale pandas columns with min-max, z-score, robust and log scaling, normalize within groups, turn values into shares and indexes, and avoid data leakage.

Columns rarely share a scale. A person's age is between 0 and 100, their income may run to six figures, and a rating is 1 to 5. If you plot them on one axis, compare them, add them up or feed them to a distance-based algorithm such as k-nearest neighbours or k-means, the biggest numbers dominate simply because they are bigger. Normalizing (or scaling) puts columns on comparable scales without changing what each column says about the ranking of its values.

You do not need a machine-learning library for this. pandas can do all the common methods in one line, which also makes the formulas easy to understand. (If you later build a model pipeline, the Data Science course shows the scikit-learn versions.)

The data

We will use a small table of cities. Notice the very different ranges and the one extreme value:

Python Columns on different scales

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Population ranges up to nine and a half million while the rating never leaves 3.5 to 4.7. Any calculation that mixes them is dominated by population.

Min-max scaling: squeeze into 0 to 1

Min-max scaling maps the smallest value to 0 and the largest to 1:

scaled = (x - min) / (max - min)

It keeps the shape of the distribution. Everything lands in a fixed range, which is handy for charts and for algorithms that expect inputs between 0 and 1. The weakness: one extreme value squeezes all the others toward 0.

Python Min-max scaling in one line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the population result: five of the six cities sit below 0.07, because Faro stretches the range. The scaling is correct, but it is not very informative when there are outliers.

Z-score standardization: mean 0, spread 1

The z-score says how many standard deviations a value is above or below the column's mean:

z = (x - mean) / std

After standardization the column has mean 0 and standard deviation 1. Positive means above average, negative below. The scale has no fixed limits, so outliers stay visible instead of being squeezed. It is the usual choice for statistics and many models.

Python Z-scores and a sanity check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pandas' std() uses ddof=1 (the sample standard deviation), while NumPy's np.std uses ddof=0 by default. Either choice is fine as long as you are consistent. Scikit-learn's StandardScaler uses ddof=0, so numbers differ slightly from the pandas version.

Robust scaling: median and IQR

When outliers are present, the mean and standard deviation are pulled toward them. Robust scaling uses statistics that resist outliers: the median for the centre and the interquartile range (IQR, the distance from the 25th to the 75th percentile) for the spread:

robust = (x - median) / IQR

Python Robust scaling resists the outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With z-scores the five small cities look nearly identical (all around -0.4), because Faro inflates the standard deviation. With robust scaling they spread out and Faro is clearly an extreme (a value above 20). Use robust scaling when you cannot or do not want to remove the outliers first (see Duplicates and Outliers).

Log scaling for skewed data

Values such as population, income and web traffic are right-skewed: many small values and a few giant ones. Multiplicative differences matter more than additive ones, so take the logarithm. np.log1p(x) computes log(1 + x), which also works for zeros:

Python Log transform, then scale

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the log scale the cities spread from 4.3 (21,000 people) to 7.0 (9.5 million), so min-max scaling now puts them across the whole 0 to 1 range.

Which method should I use?

Method Formula Use when
Min-max (x - min) / (max - min) You need a fixed 0 to 1 range and have no big outliers
Z-score (x - mean) / std Roughly bell-shaped data; statistics; many models
Robust (x - median) / IQR Outliers you cannot remove
Log np.log1p(x) Right-skewed, multiplicative data (money, counts, sizes)
Max-abs x / x.abs().max() Sparse data, to keep zeros at zero

Normalizing within groups

Sometimes a value only makes sense relative to its group: a score compared with others in the same class, a price compared with the same product category. Combine groupby with transform, which returns a result with the same shape as the original (see agg and transform):

Python Z-scores inside each group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A score of 85 in class A is the best of that class, even though 85 is below every score in class B.

Other useful "normalizations"

Several everyday calculations are also scaling:

  • Share of total: df["sales"] / df["sales"].sum() makes the column sum to 1 (or * 100 for percent).
  • Index to a base period: prices / prices.iloc[0] * 100 shows growth relative to the first row (the first value becomes 100).
  • Clip: s.clip(lower=s.quantile(0.01), upper=s.quantile(0.99)) limits extreme values instead of removing them.
  • Rank: s.rank(pct=True) replaces values by their percentile, which is immune to outliers.
Python Share of total and indexing to 100

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Do not leak: learn the numbers from training data only

If your goal is a predictive model, one rule matters more than any formula: compute the minimum, maximum, mean or standard deviation only on the training rows, then apply those same numbers to the test rows. If you scale the whole table at once, information about the test set leaks into training, and the evaluation looks better than it really is.

Python
train_mean = train["rent"].mean()
train_std = train["rent"].std()

train["rent_z"] = (train["rent"] - train_mean) / train_std
test["rent_z"] = (test["rent"] - train_mean) / train_std   # same numbers!

Also keep the numbers (or the fitted scaler) so you can apply the same transformation to new data later. And remember: do not scale ID columns, dates or categorical codes; scaling is for measurements.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which formula is min-max scaling?

    Min-max maps the smallest value to 0 and the largest to 1. The others describe z-score, robust scaling and share of total.
  2. 2. After z-score standardization, what are the column's mean and standard deviation?

    Z-scores are centred on zero and measured in standard deviations, so the mean is 0 and the standard deviation is 1.
  3. 3. A column has one huge outlier and you cannot remove it. Which scaling is the most sensible?

    The median and IQR are barely affected by a single extreme value, while min and max (and mean and std) are.
  4. 4. How do you compute a z-score inside each group?

    transform returns a value for every original row, so the arithmetic lines up with the original table.
  5. 5. Why should the scaling statistics be computed on training data only?

    Using the test rows to compute mean or max lets information about them influence training, which makes evaluation results look better than they are.

Summary

  • Normalizing puts columns on comparable scales so that large numbers do not dominate.
  • Min-max gives a 0 to 1 range; z-score gives mean 0 and std 1; robust scaling uses median and IQR; log handles skewed data.
  • Outliers squeeze min-max and distort z-scores; prefer robust scaling or a log transform first.
  • groupby(...).transform(...) normalizes within groups; rank(pct=True) gives percentiles.
  • Shares of total and indexing to 100 are everyday forms of scaling.
  • Compute scaling statistics on training data only, and reuse them for new data.

Next: Duplicates and Outliers is the chapter's other cleaning topic, and Finding Correlation uses scaled columns to compare relationships.