Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 64 ยท Statistics and Analysis

Exploratory Data Analysis Workflow

A repeatable pandas checklist for exploring new data: shape, dtypes, missing values, distributions, duplicates, outliers, correlations and a profile function.

You have just received a new dataset. Before you calculate a single result for a report, you should look at it. Exploratory data analysis (EDA) is that first look: a short, systematic pass that tells you what the table contains, what is broken, and what questions it can actually answer. Skipping it is how analysts end up reporting averages that include test orders, negative prices or the same row twice.

The good news is that EDA does not need cleverness, only a habit. In this lesson you will build a checklist you can run on any DataFrame, apply it to a small order table with deliberately planted problems, and finally wrap the first-look tasks into a reusable profile() function. The tools come from earlier lessons: DataFrame Basics, Missing Data and Descriptive Statistics.

flowchart TD
    A["1. Shape and types<br/>shape, info, head"] --> B["2. Missing values<br/>isna().sum()"]
    B --> C["3. Distinct values<br/>nunique, value_counts"]
    C --> D["4. Distributions<br/>describe, quantiles"]
    D --> E["5. Suspicious values<br/>duplicates, negatives, outliers, dates"]
    E --> F["6. Relationships<br/>corr, groupby"]
    F --> G["Write down findings<br/>then clean or ask the data owner"]

The first-look checklist. Each step answers one question and often sends you back to an earlier step.

The data

The example is a tiny orders export. Its problems are planted on purpose: one missing quantity, one missing status, a negative price, an absurd quantity of 500 and a duplicated row. In real life you would not know that yet; the checklist is how you find them.

Step 1 to 3: shape, types, missing values, distinct values

Python The first four questions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What each call tells you:

  • shape is the size of the problem: 11 rows and 7 columns. It is also your baseline: after every cleaning step, compare the new row count with this one.
  • info() lists each column's dtype and non-null count. Here quantity is float64 although it holds whole numbers, which is a clue that it contains missing values (an ordinary integer column cannot hold NaN). The order_date column was parsed as a real datetime because we asked for it with parse_dates; the text columns use pandas 3's str dtype.
  • head() shows a few rows so you can compare the column names with their content. Also glance at tail() and a random sample(), because files are often sorted and the first rows are not representative.
  • isna().sum() counts missing values per column. One missing quantity and one missing status; nothing else.
  • nunique() reveals the character of each column. order_id has 10 distinct values for 11 rows, which should be impossible for an identifier. country has just three values, so it is a categorical column, and status has three.

An identifier that is not unique is the first red flag of this dataset. Keep a list of such findings as you go, so you can decide later which ones to fix.

Step 4 to 6: distributions, suspicious values, relationships

Python Hunting for trouble

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Go through the describe() table like a detective:

  • quantity has a median of 2 but a maximum of 500 and a mean of 51.8. A mean that is 25 times the median means one or two giant values. The IQR rule (anything above q3 + 1.5 * IQR) flags order 1008 as an outlier. Is it a typo for 5? A wholesale order? Only the data owner can tell you; your job is to ask, not to delete silently.
  • unit_price has a minimum of -5.00. A negative price may be a refund line or a data-entry error. Note that it belongs to a row whose status is returned, which hints at the first explanation.
  • value_counts(dropna=False) includes the missing status in the count. Without dropna=False it would be invisible.
  • duplicated() finds one exact duplicate row, and order_id.duplicated() agrees. Remember that "same id, different values" would be a different and worse problem than an exact copy.
  • The date range (January 3 to January 10) is plausible. Ranges that start in 1970 or end in the future point to broken parsing.
  • The correlation matrix shows that revenue is mostly explained by quantity (0.80), which is mostly the effect of the outlier. A correlation computed on dirty data is a trap, so correlation comes last in the checklist.

Write findings down as you go, ideally as a list in a comment or notebook cell: "id 1010 duplicated", "price -5 on a returned order", "quantity 500 suspicious, ask sales". A clean-up script without such a list is guesswork, and next month's you will not remember why a row was removed.

A reusable profile function

You will repeat the early checklist on every dataset, so turn it into a function. The version below returns one row per column with its type, missing counts, number of distinct values and, for numeric columns, the minimum, median and maximum. A function that takes a DataFrame and returns a DataFrame is also easy to test, which matters later in Building Data Pipelines.

Python profile(): a one-call overview

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The profile immediately shows that temp has a minimum of -40.0 next to a median of around 12, a very suspicious value for Oslo or Rome in February, and that city has one missing entry. The function is a start; you can extend it with the most frequent value for text columns, the share of the top category, or a flag when distinct == len(df) (a candidate key).

How far should EDA go?

Match the effort to the stakes. For a quick one-off question, the first four steps may be enough. For data that feeds a dashboard, a model or a financial report, run the full checklist and keep the findings. The goal is never a prettier dataset; it is to know the dataset's limits before you rely on its answers.

Do not "fix" while you explore. It is tempting to drop the duplicate and fill the missing quantity right away. Explore first, write down what you find, and clean in a separate, repeatable step (Data Cleaning in Practice covers it). Mixing the two makes it impossible to say afterwards what you changed.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. df.info() shows that an integer-looking quantity column is float64. What is the most likely reason?

    A classic int64 column cannot store NaN. When missing values are present, pandas stores the numbers as float64 unless you convert to a nullable integer dtype.
  2. 2. order_id has 10 distinct values in 11 rows. What does this tell you?

    An identifier should have exactly one distinct value per row. Check duplicated() on the whole row and on the id to see which kind of problem it is.
  3. 3. Why is df['status'].value_counts(dropna=False) useful in a first look?

    By default value_counts leaves out missing values. dropna=False counts them as their own entry, so they cannot hide.
  4. 4. A column has median 2 and mean 51.8. What should you suspect?

    A mean far above the median signals a long right tail or outliers. Look at the maximum and at the largest few rows.
  5. 5. When in the EDA checklist should you interpret correlations?

    A single outlier can create or hide a correlation. Look at the quality of the data before you read relationships from it.

Summary

  • Explore before you analyze: shape, dtypes, missing values, distinct values, distributions, suspicious values, relationships.
  • shape, info(), head()/tail()/sample(), isna().sum(), nunique() and describe() answer most of the first questions.
  • Look for non-unique ids, impossible values (negative prices), outliers (IQR rule), duplicates, odd date ranges and missing categories.
  • Record every finding; do not clean while exploring.
  • Wrap the repeatable part in a function such as profile(df) that takes a DataFrame and returns a DataFrame.

Next: Data Cleaning in Practice turns the problems you found into a repeatable cleaning pipeline.