Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 57 ยท Projects

Project: Exploratory Data Analysis

A hands-on EDA project: summarize a messy customer dataset, handle missing values and outliers, compare groups, test relationships and write up findings.

Exploratory data analysis (EDA) is what you do before any formal model or test: look at the data, find out what is in it, spot the problems, and form questions worth testing. In this project you will explore a synthetic dataset of 500 online-store customers. You will clean it, describe it, compare groups, test two relationships, and finish with a short written summary like an analyst would hand to a manager.

The dataset is generated with a fixed seed, so every number is reproducible. In your own work you would load a CSV with pd.read_csv.

The questions

  1. Is the data clean enough to use? (Missing values, impossible values, duplicates.)
  2. What does a typical customer look like, and how spread out are they?
  3. Do customers on the loyalty plan spend more?
  4. Is spending related to age or to visits per month?
flowchart LR
  A["Load and inspect"] --> B["Clean: missing, duplicates, impossible values"]
  B --> C["Describe: center, spread, shape"]
  C --> D["Compare groups"]
  D --> E["Relationships and tests"]
  E --> F["Write up findings and limits"]

Step 1: build and inspect the data

Python Create and inspect the dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at describe(): a maximum age of 250 is impossible, and a maximum spend of 9,999 against a median near 160 deserves a look. Always check the minimum and maximum of every numeric column.

Step 2: clean, and document each decision

There is no single right way. The important thing is to make each choice deliberately and record it.

  • Duplicates: drop exact duplicate rows.
  • Impossible values: age 250 is a data-entry error; set it to missing.
  • Missing values: for 5% missing ages and no sign of a pattern, we can drop those rows for age-based analyses (or impute the median). Check why data is missing: if missingness relates to the outcome, dropping rows biases the results.
  • Outliers: do not delete values just because they are extreme. Investigate: a 9,999 order might be a genuine bulk purchase or a typo. Here we flag it and report results with and without it.
Python Clean the data and compare mean vs median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single wild value moves the mean by about 20 dollars but the median hardly at all. That is why the median and IQR are the safer summary for skewed money data, and why the spend distribution (right-skewed, with a long upper tail) is better described with them.

Step 3: compare groups and test

We now compare spending between loyalty members and others. Because spending is skewed, we use a Welch t-test on a log scale and a Mann-Whitney test as a cross-check, and report an effect size with a confidence interval (the habit from Multiple Testing and Effect Sizes).

Python Does loyalty go with higher spending?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: relationships

Correlation and a simple regression tell us how age and visit frequency relate to spending (see Correlation and Covariance and Multiple Linear Regression). Visits and age are correlated with spend in our simulation; the regression separates their effects.

Python Correlations and a regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 5: write the summary

An analysis is only useful once someone else can read it. A good summary has these parts:

  • Data and cleaning: 500 customers; 5 duplicate rows removed, 1 impossible age set to missing, 25 missing ages (5%), 1 extreme order excluded after inspection.
  • Findings with numbers: loyalty members spend about 22 to 23 dollars more on average, with a confidence interval and a moderate effect size; each extra visit per month goes with roughly 12 dollars more; age adds about 0.6 dollars per year.
  • Limits: observational data, so loyalty status might attract big spenders rather than make them (self-selection); results depend on the cleaning decisions; the data is simulated.
  • Next steps: run an A/B test of a loyalty offer (A/B Testing) to test causation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A customer's spend of 9,999 is far above all others. What should you do first?

    Outliers may be true extremes or errors. Investigate, decide deliberately, and show how conclusions change.
  2. 2. Why prefer the median over the mean for skewed spending data?

    A few very large values pull the mean upward, while the median depends only on the middle of the ordered data.
  3. 3. Loyalty members spend more. Why can't you conclude loyalty causes higher spending?

    Without random assignment, group differences can reflect who joins rather than the effect of membership.
  4. 4. Which is NOT a good EDA habit?

    Blind deletion can remove real information and bias results. Investigate and document instead.

Summary

  • EDA starts by inspecting types, missing values, duplicates, impossible values and extremes.
  • Make cleaning decisions deliberately and document them; test sensitivity to outliers.
  • Describe skewed variables with the median and IQR; compare groups with an effect size and an interval.
  • Regression separates the effects of several variables; observational data limits causal claims.
  • Finish with a short written summary with numbers, limits and next steps.

Next: Project: Analyzing an A/B Test