Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 57 ยท Practice and Review

Common Statistical Mistakes

Twelve common statistics errors, from misreading p-values and confounding to Simpson's paradox, survivorship bias and the base-rate fallacy, with demonstrations.

Most statistical errors in the real world are not arithmetic slips. They are errors of reasoning: answering the wrong question, trusting a number out of context, or letting a tempting story replace evidence. This final lesson walks through twelve of the most common mistakes, shows a few of them in code so you can see them happen, and gives you a checklist to carry into your own work. Read it as a review of the whole course.

Mistakes in interpreting results

1. Reading the p-value as the probability the null is true. A p-value of 0.03 means: if there were no effect, data this extreme would occur about 3% of the time. It does not say there is a 3% chance the null is true, nor a 97% chance the effect is real. To get probabilities of hypotheses you need a prior (Bayesian methods).

2. Treating "not significant" as "no effect". A p-value above 0.05 can arise because there is no effect, or because the sample was too small to detect one. Look at the confidence interval: a wide interval that includes both zero and large effects says "we don't know".

3. Confusing statistical and practical significance. With enough data, trivial differences become significant. Always report the effect size and ask whether it matters.

4. Misreading confidence intervals. A 95% interval is a statement about the procedure: 95% of intervals built this way would capture the truth. It is not a range containing 95% of the data.

5. Mistaking correlation for causation. Observational association can come from confounding, reverse causation or chance. Only a randomized experiment, or very careful causal reasoning, supports a causal claim.

Mistakes in design and data

6. Biased samples. A survey of people who answer an online poll tells you about people who answer online polls. Selection bias, non-response bias and survivorship bias all make a sample unrepresentative. A famous example: studying only the bombers that returned from missions and reinforcing the damaged spots, when the planes hit elsewhere never came back.

7. Ignoring the base rate. A rare event stays rare even after a good test. This is the problem from the medical-test example in the practice set; the code below shows it again with new numbers.

8. Simpson's paradox. A trend present in every subgroup can reverse when the groups are combined, if group sizes are unbalanced. Always ask whether a lurking grouping variable changes the story.

Python Simpson's paradox: which treatment is better?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treatment A has the higher success rate in both mild and severe cases, yet treatment B looks better overall. The reason: treatment A was given mostly to the severe cases, which are harder to treat. The grouped analysis is the right one here, because severity influences both who gets which treatment and the outcome.

Python The base-rate fallacy by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even a test that is right 95% to 99% of the time yields positives of which only about 3.7% are real, because healthy people are so much more numerous. Whenever a "positive" signal looks alarming, ask how common the condition is.

Mistakes in analysis

9. Testing many things and reporting the hits. Every extra comparison adds a chance of a false positive. Peeking at an A/B test repeatedly is the same mistake in time. Preregister the main hypothesis, correct for multiplicity and mark exploratory findings as exploratory.

Python Mining noise for 'significant' results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Report the number of tests you ran. A single hit among forty is exactly what chance predicts.

10. Overfitting. A model with enough flexibility will fit the noise in your data and fail on new data. Evaluate on held-out data, prefer simpler models, and be suspicious of Rยฒ near 1 on small samples.

11. Extrapolating beyond the data. A regression is only reliable within the range of the predictors you observed. A line fitted to ages 20 to 60 says nothing trustworthy about age 120 or age 2.

12. Misusing averages and ignoring spread. Reporting only a mean hides skew, outliers and variability. Two groups with the same mean can look very different; and the "average person" often does not exist. Show distributions and report the median, quartiles or a plot along with the mean.

A pre-flight checklist

Before sharing any analysis, ask yourself:

  1. Question: What exactly am I trying to learn, and was it decided before I saw the data?
  2. Data: How were the data collected? Who is missing? Are there errors, duplicates, outliers?
  3. Design: Is there randomization? What are the possible confounders?
  4. Method: Are the assumptions reasonable (independence, shape, equal variance, linearity)? Did I check them with plots?
  5. Multiplicity: How many things did I test or try? Did I adjust?
  6. Size: What is the effect size, with an interval? Does it matter in practice?
  7. Generalization: Does the model work on new data? To which population do the results apply?
  8. Communication: Would a reader understand what I did, what I found and what I am unsure about?

Where to go next

You now have a foundation that connects probability, descriptive statistics, inference, regression, Bayesian thinking and computer-based methods. Natural next steps: build a few projects on real datasets (public data portals have thousands), learn a statistics-heavy library such as statsmodels in depth, explore machine learning, which is regression and classification with an emphasis on prediction, or study causal inference to learn when observational data can support causal claims. Revisit the track overview at Probability and Statistics any time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A study finds p = 0.20 for a drug effect with only 12 participants. The best interpretation is:

    Non-significance is not evidence of absence, especially with a small sample. The confidence interval shows which effect sizes remain plausible.
  2. 2. In Simpson's paradox, what explains the reversal of the trend?

    When group sizes are unbalanced and the grouping variable affects the outcome, the combined trend can differ from every subgroup trend.
  3. 3. You test 40 unrelated predictors against noise at alpha = 0.05. About how many 'significant' results do you expect?

    5% of 40 is 2 false positives expected under the null.
  4. 4. Which practice most directly protects against overfitting?

    Out-of-sample evaluation reveals whether the model learned real structure or just noise.
  5. 5. A rare condition (0.2% prevalence) is tested with a 95%-specific test. A random positive is probably:

    With low prevalence, false positives from the large healthy group outnumber true positives, even for an accurate test.

Summary

  • Many errors are about reasoning: misreading p-values and intervals, treating non-significance as proof, and confusing correlation with causation.
  • Design flaws (biased samples, confounding, survivorship bias) cannot be repaired by clever analysis.
  • Watch for Simpson's paradox, the base-rate fallacy, multiple testing, overfitting and extrapolation.
  • A checklist covering question, data, design, method, multiplicity, effect size, generalization and communication keeps analyses honest.

Back to the track: Probability and Statistics