Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 76 ยท Exploratory Data Analysis

A Systematic EDA Process

Learn a repeatable exploratory data analysis routine: look at the table, then single variables, pairs and groups, record findings and avoid fishing expeditions.

Exploratory data analysis (EDA) is the stage where you get to know a dataset before you model it or report on it. You look at distributions, relationships and oddities, and you form questions. A good EDA catches problems that would quietly ruin later steps: a column that is mostly empty, a price recorded in two currencies, a spike that is really a data-entry error.

Beginners often explore by wandering: plot something, then something else, whatever looks interesting. That works for small tables and fails for large ones, and it makes it easy to fool yourself. This lesson gives you a process: a fixed order of passes, a checklist of questions, a habit of writing findings down, and a guard against fishing for patterns. The following lessons zoom into each pass: one variable, relationships and time.

flowchart LR
    A["1. First look<br/>shape, types, missing"] --> B["2. Univariate<br/>one column at a time"]
    B --> C["3. Bivariate<br/>pairs of columns"]
    C --> D["4. Multivariate<br/>groups and combinations"]
    D --> E["5. Record and decide<br/>findings, next steps"]
    E -.->|"new questions"| B

EDA moves from simple to complex, and every pass can send you back with a new question.

Before you plot: know the question

Return to the question you framed (Framing a Good Question). EDA is not aimless: it asks whether the data can answer that question, which columns matter, and what must be cleaned first. Keep the question in a comment at the top of your script.

Pass 1: the first look

Before any chart, answer the dull questions. They take two minutes and prevent most surprises.

  • How many rows and columns? Is one row one order, one customer, one day?
  • What are the types? Are dates real dates, are numbers stored as text?
  • How much is missing, and is it spread evenly?
  • Are there duplicate rows or duplicate ids?
  • What is the time range, and are there gaps?

If you wrote the profiling function, run it now.

Pass 2: univariate (one variable at a time)

For each column on its own: numeric columns get a histogram and summary numbers (center, spread, minimum, maximum, extreme values); categorical columns get counts and the share of the biggest group. You are checking for impossible values, heavy skew, strange spikes and categories that should be merged. The next lesson covers this pass in depth.

Pass 3: bivariate (pairs)

Now look at two columns together: numeric against numeric (scatter plot, correlation), numeric by category (grouped summaries, box plots), category against category (cross-tabs). Prioritize pairs that involve the outcome you care about. See Relationships Between Variables.

Pass 4: multivariate (combinations)

Real effects often hide behind a third variable. A relationship may reverse once you split by region, or vanish once you account for time. Split key plots by one extra category, or compare correlation matrices across groups. This pass is also where you notice that two columns carry the same information.

Pass 5: record and decide

At the end of each pass, write down what you saw, in plain sentences, with the number. For example: "6% of units are missing, almost all in the North region, so they are probably not random." Findings become a to-do list: fix, drop, transform, collect more data, or ask the data owner.

A worked example on synthetic sales data

Here is an entire first EDA on a small, made-up sales table. The data contains a few deliberate flaws (missing values, a duplicate row, an impossible discount) so that the checks have something to find.

Python Pass 1: the first look

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first look already reports four things worth writing down: 30 missing units, one duplicate row, a maximum discount of 1.8 (impossible), and prices and revenue that look plausible. You have not drawn a single chart yet.

Python Passes 2 to 4 with figures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What do the charts say? Revenue per order is skewed with a long right tail (a few big orders). Desks dominate revenue because they cost most per unit, and the regional differences are small by comparison, so the product effect is much stronger than the region effect. Each of those is a sentence for your findings log, together with the number that supports it.

The EDA checklist

Copy this list into your project and tick items off.

  1. What is one row? What is the time range?
  2. Types correct? Dates parsed? Numbers stored as numbers?
  3. Missing values: how many, where, why?
  4. Duplicates: whole rows and key columns.
  5. Impossible or suspicious values: negative ages, future dates, 180% discounts.
  6. Each numeric column: shape of the distribution, extreme values.
  7. Each categorical column: how many levels, rare levels, spelling variants.
  8. Outcome versus each candidate predictor.
  9. Do relationships change by group or over time?
  10. Which columns are redundant?
  11. What would I need to ask the data owner?

Keep a findings log

Create a plain text list, one line per finding, each with the evidence and the action:

# Finding Evidence Action
1 units missing for 5% of rows 30 of 601 Check source; impute or flag
2 One discount above 100% max = 1.8 Remove as an entry error
3 Revenue is right-skewed median below mean Use median; consider log scale

A log keeps you honest, lets a colleague retrace your thinking, and becomes the skeleton of your final report (Communicating Results to Non-Experts).

Avoiding fishing expeditions

If you look at 50 relationships, a few will look interesting by pure chance. That is not a discovery; it is a lottery. To stay honest:

  • Separate exploring from confirming. Patterns found while exploring are hypotheses. Test them on fresh data or with a pre-planned test (Hypothesis Testing Done Right).
  • Count what you looked at. If you checked 40 pairs, say so when you report the one that stood out.
  • Prefer explanations that make sense. A relationship with a plausible mechanism deserves more trust than one that popped out of a random scan.
  • Hold out data. If the dataset is large, set aside a part you never explore, and use it to check your final claims.

Exploring is not wrong; presenting an exploratory finding as if it had been planned in advance is. Label it for what it is.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the best first step of an EDA?

    The first look reveals structural problems cheaply. Everything later depends on knowing what a row is and whether the types are right.
  2. 2. Which pass compares revenue across products?

    Two variables are involved (a number and a category), so it is a bivariate pass. Adding a third, such as region, would make it multivariate.
  3. 3. Why record findings in a log?

    A log with evidence and actions lets others retrace your reasoning and prevents you from forgetting what you decided.
  4. 4. You scan 60 pairs of columns and find one with a striking correlation. How should you treat it?

    With many comparisons some will look strong by chance. Treat it as a lead, not a result, and say how many things you examined.
  5. 5. A relationship that flips direction after splitting by region shows the value of which pass?

    Looking at combinations of variables can reveal that a third variable changes or hides a relationship.

Summary

  • EDA is a process, not wandering: first look, univariate, bivariate, multivariate, then record and decide.
  • Start from your question, and always check shape, types, missing values, duplicates and impossible values first.
  • Write findings with evidence and a next action in a log.
  • Exploring creates hypotheses; confirm them with fresh data or a pre-planned test, and say how much you looked at.
  • A checklist makes EDA fast and repeatable on every new dataset.

Next: Looking at One Variable at a Time, the second pass in detail.