Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 39 ยท Data Preprocessing

Data Cleaning

Clean data before mining: handle missing values, smooth noise with binning, remove duplicates, fix inconsistent labels and treat outliers, using pandas examples.

"Garbage in, garbage out" is the oldest rule in data mining. Real data has holes, typos, duplicates and values that are physically impossible, and algorithms do not complain: they quietly learn from the mess. Data cleaning (also called data cleansing or scrubbing) is the first preprocessing step. You detect problems, decide what each one means, and repair or remove it. This lesson walks through the four classic problems: missing values, noise, duplicates and inconsistency.

Missing values

Values go missing for different reasons, and the reason matters:

  • MCAR (missing completely at random): a sensor dropped some readings by chance. Safe to drop or fill.
  • MAR (missing at random): missingness depends on other columns (older customers skip an online form). Fill using those columns.
  • MNAR (missing not at random): the value is missing because of what it would be (high earners leave income blank). Filling naively biases results; the missingness itself may be a feature.

Common remedies:

  1. Drop rows with missing values: simple, but wastes data and can bias the sample.
  2. Drop columns that are mostly empty.
  3. Fill with a constant (like "unknown", or 0 when zero is truthful).
  4. Fill with a statistic: mean (symmetric data), median (skewed data), mode (categories).
  5. Fill using related rows: group means, nearest neighbors, or a regression model.
  6. Add an indicator column "was missing", then fill.
Python Five ways to handle missing values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fill with the group mean (dept B earns more than A) instead of the overall mean when groups differ. For a model-based fill, scikit-learn offers KNNImputer and IterativeImputer, covered in Handling Missing Values. Whatever you choose, learn the fill values on training data only and reuse them on new data.

Noise and binning

Noise is random error in a measured value: a thermometer that jitters, a typo in an amount. Smoothing reduces it. Simple methods are:

  • Binning: sort values, split into bins, replace each value by its bin mean, median or boundary.
  • Regression: fit a function and use fitted values.
  • Moving average for ordered data.
  • Outlier removal after clustering or statistical tests.
Python Smoothing by bin means

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Smoothing by means makes values within a bin identical, which hides small detail but removes jitter. Boundary smoothing snaps each value to the nearer edge of its bin. Pick the bin count by looking at the data: too few bins flatten real structure.

Duplicates

Duplicates arise when the same entity is entered twice, when sources are merged, or when a file is loaded twice. They inflate counts and can leak between training and test sets, which makes a model look better than it is. Exact duplicates are easy to drop; near duplicates ("Jon Smith" and "John Smith") need normalizing and fuzzy matching.

Python Exact and near duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inconsistent data

Typical inconsistencies are mixed spellings (NY, New York), mixed units (km and miles), mixed date formats, stray whitespace and impossible combinations (a birth date after a hire date). Fix them with a mapping table and validation rules.

Python Standardize labels and check rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Outliers

An outlier may be an error (a height of 1,800 cm) or a rare but real event (a billionaire customer). Do not delete blindly. Look first, ask the data owner, then choose: correct, cap (winsorize), remove, or keep and use a robust method. A common rule flags values outside Q1 - 1.5 * IQR and Q3 + 1.5 * IQR; the anomaly detection chapter later in the course goes much further.

Python The IQR rule and capping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A cleaning workflow

  1. Profile: counts, missing share, ranges, unique values (Data Types and Data Quality).
  2. Decide per problem with the domain: fix, fill, flag or drop.
  3. Write the steps as code, never by hand in a spreadsheet, so they can be rerun on tomorrow's data.
  4. Keep the raw data untouched and log what changed and how many rows it affected.
  5. Recheck after cleaning.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. High earners leave the income field blank because it is high. This is:

    Missing not at random: the chance of being missing depends on the missing value itself.
  2. 2. Which fill value is usually safest for a skewed numeric column?

    The median is not pulled by extreme values, unlike the mean.
  3. 3. Smoothing by bin means replaces each value in a bin by:

    All values in a bin become the bin's average, reducing random noise.
  4. 4. Why can duplicates make a model look better than it is?

    If a copy of a training row is in the test set, the model is tested on something it already saw.
  5. 5. What should you do with an outlier first?

    Outliers can be errors or valuable signals; investigate before deciding.

Summary

  • Missing values: understand the mechanism (MCAR, MAR, MNAR), then drop, fill, or flag.
  • Noise is reduced by binning, regression or moving averages.
  • Remove exact duplicates and normalize text to find near duplicates.
  • Standardize labels and units; validate with rules.
  • Investigate outliers before removing; use the IQR rule as a first flag.
  • Script every step and keep the raw data.

Next: Data Integration and Reduction