Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 76 ยท Cleaning and Preparing Data

Outliers, Errors and Impossible Values

Detect outliers with IQR and z-score rules, catch impossible values with domain rules, and choose between keeping, fixing, winsorizing or removing them.

An outlier is a value far from the rest. That is all the word means. It does not say whether the value is a mistake, a rare but real event, or the most interesting row in the table. A salary of 310,000 among salaries near 50,000 might be a typo, a CEO, or a genuine senior engineer. Deleting it blindly can erase the very thing you are looking for; keeping it blindly can distort every average you compute.

This lesson separates three situations that often get lumped together: statistical outliers (unusual compared with the other values), impossible values (breaking a rule of the real world, such as an age of -3), and real extremes (unusual but true). You will learn the two standard detection rules, why one of them can miss outliers, how domain rules catch what statistics cannot, and what to do once a value is flagged.

Detection rule 1: the IQR fences

The interquartile range (IQR) is the distance between the 25th and 75th percentiles, so it describes the middle half of the data. The classic rule flags anything more than 1.5 IQRs below the first quartile or above the third quartile. These limits are called fences; the box in a boxplot uses the same rule. Because quartiles ignore the extremes, a few huge values do not move the fences.

Detection rule 2: the z-score

The z-score says how many standard deviations a value sits from the mean: (x - mean) / std. A common cut-off is an absolute z of 3. It is simple, but the mean and standard deviation are themselves pulled around by outliers. A huge value inflates the std, which shrinks every z-score, which can hide the outliers that caused the problem. This is called masking.

Python IQR fences vs z-score on salaries with planted errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three results are worth studying:

  1. The IQR fences caught all three planted errors (-4,000, 240,000 and 310,000) and also a fourth value, 76,567. That value is a legitimate point in the tail of a normal distribution. With 300 values, a handful of genuine points will always fall outside 1.5 IQRs, so a flag is a reason to look, not a verdict.
  2. The z-score rule flagged only two. The two huge salaries pushed the standard deviation up to about 20,500 (the real spread is 9,000), so the z-score of -4,000 is only -2.81, below the cut-off. That is masking in action.
  3. The robust z-score replaces the mean with the median and the std with the median absolute deviation (MAD). Both are barely affected by outliers, so the rule flags three values. Use this version when you expect contamination.

Tip: Look at the data before you trust a rule. A histogram or boxplot (see Looking at One Variable at a Time) shows in seconds whether the flagged values are blunders, a second population, or an ordinary long tail.

Domain rules catch what statistics miss

No amount of arithmetic can tell you that 250 is not a valid age. Domain rules come from knowing the world: ages lie between 0 and about 120, a week has 168 hours, heights are measured in centimetres or metres but not both. Such rules find errors that sit inside the normal range of a column, and they find impossible values even when they are not statistical outliers.

Use rules as flags, not as an immediate delete. A value in the wrong unit can often be repaired: 1.75 in a column of centimetres is almost surely 175 centimetres typed in metres.

Python Domain rules, impossible values and a unit repair

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row 3 breaks two rules at once (age 250 and a height of 1.75 cm), which suggests the whole row is unreliable, not just one cell. A row with several problems is a stronger candidate for removal than a row with a single repairable one. The age of 105 passed the rule: it is rare but possible, and the rule is a bound on what is impossible, not on what is usual. Counting how many rules each row breaks, as n_problems does, is a handy way to prioritize.

Keep, fix, cap or remove?

Once something is flagged, the question is what it is. A short decision path works well:

flowchart TD
    A["Value flagged"] --> B{"Impossible in the real world?"}
    B -->|"Yes"| C{"Can the true value be inferred?"}
    C -->|"Yes, e.g. wrong unit"| D["Fix and note the repair"]
    C -->|"No"| E["Set to missing, then handle as missing"]
    B -->|"No"| F{"Does it belong to your question?"}
    F -->|"Yes, a real extreme"| G["Keep; use robust methods or a log scale"]
    F -->|"No, a different population"| H["Analyze separately or exclude, with a note"]

Impossible values become repairs or missing values; possible values stay, and you adjust the method instead of the data.

Some practical points:

  • Keep real extremes. If you are studying fraud, the unusual transactions are the point. If you are forecasting revenue, a genuine big customer is revenue.
  • Setting an impossible value to missing is usually better than deleting the whole row, because the other columns may be fine. Then use the strategies from Handling Missing Values.
  • Always record what changed: which rows, which rule, which repair. Keep the original column and add a flag column so the analysis stays reviewable.
  • Do not remove outliers just to improve a model's score or to make a result significant. That is a form of cheating that only shows up when the model meets reality.

Taming extremes without deleting them

Sometimes the right response is to limit an outlier's influence rather than remove it.

Winsorizing replaces values beyond chosen percentiles (say the 1st and 99th) with the percentile value itself. The row stays; the extreme is pulled in. Robust statistics such as the median, the IQR and the trimmed mean are barely affected by extremes in the first place. A log transform compresses the right tail of skewed positive data like income or prices, which often makes extreme values look ordinary.

The experiment below starts from right-skewed incomes with a known truth, adds five data-entry blunders (values multiplied by 40), and compares the summaries.

Python Robust statistics and winsorizing under contamination

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read down the "with blunders" column. Five bad values out of 500 (1%) lift the mean from 52,162 to 78,112 and inflate the std from about 17,800 to over 265,000. Meanwhile the median (49,666 against 49,466), the trimmed mean and the IQR barely move. Winsorizing brings the mean back to about 52,970 and the std to about 20,000, though not exactly to the truth, because capping also shortens the genuine upper tail by a little. Winsorizing changed 10 rows: the 5 blunders plus 5 on the low side, which is how a symmetric rule works. Capping both ends is a trade-off, and a rule that only caps the top may fit better here.

Which tool to choose depends on the goal: robust statistics for describing, winsorizing or a log transform for feeding models that are sensitive to extremes (such as linear regression), and domain repairs for genuine errors.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can the z-score rule miss outliers that the IQR rule catches?

    Extreme values raise the standard deviation, so every z-score gets smaller. Quartiles are not pulled the same way.
  2. 2. A column of ages contains 250. Which kind of finding is this?

    No person is 250 years old, so no statistical argument is needed. The domain rule marks it as invalid.
  3. 3. A height of 1.75 appears in a column measured in centimetres. What is the most sensible action?

    When the true value can be inferred with confidence, a documented repair is better than deleting or inventing a value.
  4. 4. What does winsorizing do?

    The rows stay but the extremes are capped at the chosen lower and upper percentiles.
  5. 5. You are studying credit-card fraud and flag unusually large transactions as outliers. What is the best next step?

    Whether to remove a flagged value depends on the question. In fraud analysis, unusual events are often what you are looking for.

Summary

  • An outlier is just an unusual value; decide separately whether it is an error, an impossible value or a real extreme.
  • IQR fences are robust; plain z-scores can suffer masking, so prefer a robust z-score (median and MAD) when contamination is likely.
  • Domain rules (valid ranges, units, relationships) catch errors that statistics cannot; count how many rules each row breaks.
  • Fix what can be inferred, turn the rest into missing values, keep real extremes, and document every change.
  • Winsorizing, robust statistics and log transforms limit the effect of extremes without deleting rows.

Next: Cleaning Text and Categories.