Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 57 ยท Projects

Project: Exploratory Data Report

Build an exploratory data report in Matplotlib and pandas: histogram grid, correlation heatmap, scatter matrix, outlier highlighting and a one-page report function.

Before you model, forecast or present any dataset, you explore it. What does each column look like? Which columns move together? Are there odd rows that will distort every average? Answering those questions with charts is called exploratory data analysis, and it is where Matplotlib earns its keep: ten minutes of plots often save ten hours of confusion later.

In this project you will explore a synthetic customer dataset with pandas and Matplotlib. You will draw a grid of histograms, a correlation heatmap and a scatter matrix, flag outliers with a simple rule, and finally wrap everything in one function that turns any DataFrame into a one-page report figure. It pulls together ideas from Multiple Subplots, Heatmaps and Images, GridSpec and subplot_mosaic and Plotting NumPy and pandas Data.

The plan for the report

Distributions One histogram per column.
Correlations A heatmap of how columns move together.
Outliers Flag rows far from the crowd.
One page Wrap it all in a function.

Look at one column, then pairs, then everything at once.

1 / 2

Step 1: a synthetic customer dataset

We need data that behaves like the real thing: skewed incomes, counts of visits, a few extreme spenders. We generate it with a fixed seed, so every run produces the same 400 customers. Spending depends on income and visits plus noise, and we deliberately inflate three rows to create outliers that the report should catch.

Python Build and summarize the customers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the describe() table before you plot anything. The median of spend is about 217, yet the maximum is above 1,300: a huge gap between the middle and the top is a loud hint of outliers or a skewed distribution. The mean income is above the median too, which is typical for money. Tables tell you that something is odd; charts tell you what shape it has.

Step 2: a histogram for every column

A histogram grid answers "what does each column look like?" in one glance. Loop over the columns and the axes together with zip, draw each histogram, and mark the median with a dashed line. Using bins="fd" (the Freedman-Diaconis rule) picks a sensible bin width per column, so you do not hand-tune five different bin counts. See Histograms and Distributions for the rules.

Five columns do not fill a 2 by 3 grid. The unused sixth axes would stay as an empty box, so we hide it with set_visible(False).

Python Histogram grid with medians

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the shapes. age is roughly bell-shaped, income leans to the right with a long tail, visits is a count with a peak near 6, and tenure piles up at small values with a long tail (an exponential shape). spend shows a tall main body and a thin tail reaching far right: those are our planted customers. If you saw the same picture in real data, it would be time to ask whether those rows are errors, fraud, or your best customers.

Step 3: a correlation heatmap

Which columns move together? DataFrame.corr() returns a square table of Pearson correlations between -1 and 1. A heatmap makes it readable. Use a diverging colormap centered on zero (RdBu_r with vmin=-1, vmax=1) so that positive and negative relationships get opposite colors and zero is neutral. Print the number in every cell, and switch the text to white on dark cells so it stays legible. The ideas come from Colormaps and Color Normalization.

Python Annotated correlation heatmap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The diagonal is always 1 because every column correlates perfectly with itself. The interesting row is spend: income correlates at about 0.52 and visits at about 0.31, while tenure and age are close to zero. That matches how we generated the data, and it is the kind of statement a report should make quickly.

Correlation measures a linear relationship and is pulled around by outliers. Our three planted spenders weaken the true income-spend correlation. Always look at the scatter plot behind a number before you trust it.

Step 4: finding and highlighting outliers

How do you decide that a value is "far from the crowd"? A classic rule uses the interquartile range (IQR): compute the first and third quartiles, then call anything above Q3 + 1.5 * IQR (or below Q1 - 1.5 * IQR) an outlier. It is the same rule that decides which points a box plot draws as dots (see Box Plots and Violin Plots). It makes no assumptions about the shape of the distribution, so it works on skewed data too.

With a boolean Series is_outlier you can split the DataFrame and draw the two groups with different styles: small translucent blue dots for the crowd, larger red dots with a dark edge for the flagged rows. A scatter matrix shows every pair of columns in a grid, with a histogram on the diagonal. We keep tick labels only on the outer edges so the grid stays clean.

Python Scatter matrix with outliers in red

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rule flags five rows: our three planted customers and two real, large spenders from the random data. Notice in the spend row that the red dots sit high above the cloud for every other column; their income, age and visits look perfectly ordinary. That is a useful finding on its own: whatever makes these customers spend so much is not in these columns.

The zorder=3 argument draws the red points on top of the blue ones. Without it, a flagged point hidden behind a crowd of blue dots would be easy to miss. If you want more on ordering, see Transforms and the Artist Layer.

Step 5: one function that makes the report

Exploring once is useful; exploring the next ten datasets the same way is where a reusable function pays off. A good report function takes a DataFrame and a target column, builds one figure, and returns the figure instead of showing or saving it. The caller decides what happens next. This is the pattern from Reproducible and Automated Charts.

We lay the page out with subplot_mosaic: a KPI strip across the top, the target's histogram on the left, a bar chart of correlations below it, and a large scatter plot of the target against its strongest correlate on the right. The KPI strip is not a chart at all, just an axes with its frame switched off and some text in it.

Python A one-page report from any DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two design choices make this function reusable. First, nothing in it names a specific column except the target argument; the best predictor is found with idxmax() on absolute correlations. Second, it ends with return fig, so tomorrow you can call fig.savefig("report.png", dpi=200) for a file, or plt.show() for the screen, or put the figure into a test. Try changing the target to "income" and run it again: the whole page rearranges itself around the new column.

An exploration is only as honest as your last check. After a report like this, pick the most surprising number and look at the raw rows behind it. Charts guide your attention; the data still has the last word.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the heatmap use the RdBu_r colormap with vmin=-1 and vmax=1?

    Correlation runs from -1 to 1 with a meaningful midpoint at 0. A diverging map with symmetric limits makes zero neutral and gives the two directions of the relationship different colors.
  2. 2. A value is flagged as an outlier by the IQR rule when it lies beyond which limit?

    The IQR rule extends 1.5 interquartile ranges beyond the quartiles. It does not assume a bell-shaped distribution, which makes it a good fit for skewed data such as income or spending.
  3. 3. Why does the report function end with return fig instead of calling plt.show()?

    A function that returns the figure is reusable: the caller can show, save at any dpi, tweak it further or inspect it. Showing inside the function hides all of those options.
  4. 4. What does zorder=3 do when drawing the outlier scatter?

    Artists with a higher zorder are drawn later, so they appear above lower ones. That keeps the flagged points visible even where many typical points overlap them.

Summary

  • Exploration goes from one column (histograms) to pairs (heatmap, scatter matrix) to the whole table (the report).
  • describe() hints at skew and outliers; charts reveal their shape.
  • A diverging colormap centered on zero with printed values makes a readable correlation heatmap.
  • The IQR rule (Q3 + 1.5 * IQR) flags outliers without assuming a distribution; highlight them with a distinct color and higher zorder.
  • Hide unused axes with set_visible(False) and keep tick labels only on the outer edges of a grid.
  • Wrap a layout in a function that takes data and returns a Figure, so it can be reused, saved and tested.

Next project: Model Diagnostics Plots.