Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 111 ยท Getting Started

Exploratory Data Analysis for Machine Learning

Learn EDA step by step: check shape, types and missing values, then run univariate, bivariate and multivariate analysis with pandas, correlations and plots before modeling.

Before you train any model, look at the data. Exploratory data analysis (EDA) is the habit of asking simple questions of a data set and answering them with tables and plots: What does each column mean? How many values are missing? Are there impossible values? How is the target distributed? Which columns seem related to it? An hour of EDA regularly saves days of debugging a model that was fed broken input, and it tells you what to do in the feature engineering step.

EDA is not a separate science, just a checklist you run in the same order every time. This lesson walks through it on a small customer-churn table that the code generates, with some problems planted on purpose (missing incomes and three impossible ages), so you can see how each step finds a problem.

The EDA checklist

Shape and types Rows, columns, dtypes, missing values.
Univariate One column at a time.
Bivariate Each column against the target.
Multivariate Columns against each other.

Fix problems as you find them, and write them down.

1 / 2

Step 1: Shape, types and missing values

Start with the cheapest questions: how big is the table, what type is each column, and how much is missing? df.shape, df.dtypes, df.isna().sum(), df.nunique() and df.describe() answer all of them in a few lines. Pay particular attention to the min and max rows of describe(). They are the quickest way to spot values that cannot be right.

The table below has 600 customers and the target column churned (1 if the customer left). After the overview, the code removes the impossible ages and then moves on to the target balance and the relationships between columns.

Python EDA with pandas: overview, univariate and bivariate checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here is what each block tells you.

  • Missing values. income has 40 missing entries out of 600 (6.7%). Whether that is a problem depends on why they are missing. Before you fill them, see Handling Missing Data.
  • Impossible values. The min and max of age (-4 and 999) are not plausible ages, and the mean (42.7) and standard deviation (41.0) are distorted by them. The three rows are found and removed. In real data a value like 999 or -1 is often a code for "unknown", and the right fix is to turn it into a missing value.
  • Target balance. About 11% of customers churned. That is an imbalanced target, so accuracy will be misleading, and you will want stratified splits.
  • Univariate shape. Age and tenure are roughly symmetric (skew near 0), income has a mild right skew (0.66).

Step 2: Univariate analysis

Univariate analysis looks at one column at a time. For numeric columns, use a histogram and a box plot, and read the skew, spread and outliers. For categorical columns, count the categories and watch for rare ones, typos ("North" and "north") and categories that are almost the whole column. A column with a single value is useless to a model, and a column that is unique on every row (an ID) is useless or harmful.

Step 3: Bivariate analysis

Bivariate analysis compares two columns. The most useful pairs are each feature against the target:

  • Categorical feature against a binary target: the target rate per category, with the count beside it. Here the churn rate is 14.5% on the basic plan and 7% to 8% on plus and pro.
  • Numeric feature against a target: cut the feature into bands and compute the target rate per band, or compare box plots. Younger customers churn far more (23.5% for ages 18 to 30, about 6% for ages 40 to 50).
  • Two numeric features: a scatter plot and a correlation coefficient.

Always look at the counts, not only the rates. The region rates run from 6.4% to 16.4%, which looks like a signal. But we generated no region effect at all: with only about 150 customers per region, differences of this size are what chance produces. A table of rates without counts invites you to find patterns that are not there. When a difference matters, test it (see Descriptive and Inferential Statistics).

Step 4: Multivariate analysis and correlation

Multivariate analysis looks at many columns at once. The standard tool is the correlation matrix. Pearson correlation measures straight-line association, and Spearman correlation measures any monotonic association (it works on ranks and shrugs off outliers). In the output above, age and income correlate at 0.63 because we generated income to rise with age, while each of age, income and tenure_months has a modest negative correlation with churn (-0.18 to -0.15), and Spearman gives the same picture.

Strong correlation between two features is a warning for linear models (they carry overlapping information, see Multiple Regression). Strong correlation between a feature and the target is promising, but be suspicious if it is too strong: a column that is almost perfectly correlated with the target is often a leak, information that would not be known at prediction time (see Nested CV and Leakage). Correlation also does not capture non-linear or interaction effects, so a near-zero value is not proof that a feature is useless.

Plots: one page of pictures

Numbers hide shapes, so make a few plots for every project. The next example draws six standard views: a histogram of a skewed column and its log transform, a box plot by group, a target rate by category, a colored scatter plot and a correlation heatmap.

Python Six standard EDA plots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two panels show why analysts log-transform money columns: the raw income is stretched to the right (skew 0.75), and its logarithm is much closer to a symmetric bell. The box plot confirms ages are similar across plans, the bar chart repeats the churn-by-plan finding, and the heatmap puts the correlation table into color so that the strong cell (age and income) stands out at a glance.

EDA rules of thumb

  • Do EDA on the training data only. If you let test rows influence decisions (which outliers to cap, which features to keep), your test score becomes optimistic. Split first (Train, Validation and Test Sets), explore the training part.
  • Write down every problem and decision. A short list ("income missing 6.7%, impute with the median; ages above 100 are errors") becomes the spec for your pipeline.
  • Be skeptical of patterns. Many comparisons produce spurious ones. Check counts and treat surprising findings as hypotheses.
  • Ask the data owner. Does 999 mean unknown? Are duplicates expected? EDA finds questions, people answer them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In describe(), a column of ages has min -4 and max 999. What should you do first?

    Impossible values are data errors or placeholder codes. Find out what they mean, then remove or convert them to missing values. Scaling would only hide the problem.
  2. 2. The churn rate is 6% in one region and 16% in another, with about 150 customers each. What is the sensible reading?

    Small groups produce noisy rates. Always read target rates together with group sizes, and use a statistical test for differences you plan to act on.
  3. 3. Which correlation measure is best for a relationship that always rises but is curved?

    Spearman works on ranks, so any monotonic relationship scores high. Pearson only measures straight-line association and underestimates curved but monotonic relationships.
  4. 4. Why should EDA decisions be made on the training split only?

    If the test data shaped your cleaning or feature choices, it is no longer an unbiased check of how the model will do on new data.

Summary

  • EDA is a repeatable checklist: shape, types and missing values, then univariate, bivariate and multivariate views.
  • describe() min and max reveal impossible values; isna().sum() counts missing data; the target balance tells you which metrics to use.
  • Compare each feature with the target using target rates (with counts), bands and plots; small groups mislead.
  • Pearson measures linear association, Spearman monotonic association; a near-zero correlation does not rule out other relationships, and a near-perfect correlation with the target may be a leak.
  • Explore only the training data, and write down the cleaning decisions for your pipeline.

Next: Train, Validation and Test Sets shows how to split the data so EDA stays honest.