Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 39 ยท Data Preprocessing

Data Transformation

Transform data for mining: min-max and z-score scaling, log transforms, discretization (binning), one-hot and ordinal encoding, and feature construction with Python.

Clean data is still not always ready for an algorithm. Distance-based methods get confused when one column is in dollars and another in fractions. Rule miners want categories, not decimals. Models cannot read the word "Oslo". Data transformation converts data into a form suited to the algorithm. The main tools are normalization, discretization, encoding and feature construction.

Normalization (scaling)

Consider customers described by age (20 to 70) and income (20,000 to 200,000). In a Euclidean distance, income differences are thousands of times larger, so age is ignored. Scaling puts attributes on comparable ranges.

  • Min-max normalization maps to a range, usually 0 to 1: x' = (x - min) / (max - min). Simple, keeps shape, but a single outlier squeezes everything else.
  • Z-score standardization: x' = (x - mean) / std. Result has mean 0 and std 1. Better with outliers, and the default for many algorithms.
  • Robust scaling uses median and IQR, so extreme values matter less.
  • Decimal scaling divides by a power of 10 so all absolute values are below 1.
Python Min-max and z-score by hand and with sklearn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the min-max line: the outlier at 400,000 pushes the four normal values into the range 0 to 0.12. Robust scaling keeps them spread out. As always, fit the scaler on training data and apply it to new data; see Scaling and Normalization.

Skewed data and power transforms

Prices, incomes and counts are often right-skewed: many small values, a few huge ones. A log transform (log(1 + x)) pulls in the long tail and makes patterns easier to see. Square root and Box-Cox / Yeo-Johnson transforms do similar work.

Python A log transform reduces skewness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Discretization (binning)

Discretization turns a continuous attribute into a few intervals. Rule and tree methods often work with categories, and binning can hide noise. Methods:

  • Equal-width: bins of the same size. Easy, but outliers leave many bins empty.
  • Equal-frequency (quantile): each bin holds about the same number of values.
  • Supervised: pick cut points that best separate the classes (for example with a decision tree).
  • Concept hierarchies: use domain knowledge (age 0-17 child, 18-64 adult, 65+ senior).
Python Equal-width, equal-frequency and domain bins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Encoding categorical attributes

Most algorithms need numbers.

  • Label (ordinal) encoding: map categories to integers. Right for ordered categories (S < M < L); misleading for nominal ones, since it implies Rome > Oslo.
  • One-hot encoding: one binary column per category. Safe for nominal data but widens the table, a problem for columns with thousands of values.
  • Binary / hashing / target encoding: compact alternatives for high-cardinality columns. Target encoding must be done inside cross-validation to avoid leakage.
Python Ordinal versus one-hot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Feature construction and aggregation

Often the best gain comes from new attributes derived from old ones: price_per_unit, days_since_last_purchase, is_weekend, ratios, differences, counts per customer. Domain knowledge beats algorithm tuning here.

Python Constructing features from transactions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This kind of aggregation turns many rows per entity into one row per entity, the shape mining algorithms expect. It leads straight into the distance measures of the next lesson.

Choosing a transformation

Which transform for which algorithm?

Distance based Scale: k-NN, k-means, PCA.
Tree based Scaling not needed.
Rule mining Discretize and one-hot.
Linear models Scale, log skewed data.
1 / 2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Min-max normalization of the value 50 in the range 0 to 200 gives:

    (50 - 0) / (200 - 0) = 0.25.
  2. 2. Which scaler is least affected by a single extreme outlier?

    It uses the median and IQR, which barely move when one value is extreme.
  3. 3. Why is label-encoding a nominal city column risky?

    Integers suggest Rome is bigger than Oslo, which can mislead distance-based or linear models.
  4. 4. Equal-frequency binning gives bins that:

    Quantile bins balance counts, not widths.
  5. 5. A log transform is most useful for:

    It compresses the long right tail.

Summary

  • Scale numeric attributes for distance-based methods: min-max, z-score or robust.
  • Log or power transforms tame skewed data.
  • Discretize with equal-width, equal-frequency, supervised or domain-based bins.
  • Encode categories: ordinal for ordered, one-hot for nominal.
  • Construct new attributes; fit all transformers on training data only.

Next: Similarity and Distance Measures