Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 57 ยท Projects

Project: Model Diagnostics Plots

Plot a scikit-learn classifier with Matplotlib: decision boundary, confusion matrix, ROC and precision-recall curves, feature importance and a learning curve.

A single accuracy number hides almost everything that matters about a classifier. Does it fail evenly on both classes, or does it miss one of them? Is it confident near the boundary or guessing? Is it overfitting, and would more data help? Diagnostic plots answer these questions, and most of them are a few lines of Matplotlib on top of numbers that scikit-learn already computes.

In this project you will train classifiers on synthetic data and build a small diagnostics kit: a decision boundary drawn with contourf, a confusion matrix heatmap, ROC and precision-recall curves, a feature-importance chart with error bars, and a learning curve with confidence bands. You do not need to know how the models work inside; you need to know how to read what they produce. The plotting tools are from Heatmaps and Images, Error Bars and Uncertainty and Statistical Plots Recipes.

Five questions, five plots

Boundary Where does the model switch its answer?
Confusion matrix Which mistakes does it make?
ROC and PR How good at every threshold?
Importance Which features matter?

Plus a learning curve to see if more data would help.

1 / 2

Step 1: data, a model and a confusion matrix

We generate a binary classification problem with make_classification: 600 examples, 6 features of which only some carry signal, and 5% of labels flipped to make life realistic. We hold out 30% for testing and fit a random forest. Always diagnose a model on data it did not train on; the training score is flattering.

A confusion matrix counts predictions in four groups: true negatives, false positives, false negatives and true positives. We draw it with imshow and write the counts into the cells, so the picture is the table.

Python Train a forest and draw its confusion matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first printout is the first diagnosis: the forest scores about 0.98 on the training data but about 0.87 on the test data. A gap like that says the model has partly memorized its training examples. In the heatmap, rows are the true classes and columns the predictions, so the diagonal holds the correct answers. Writing "percent of row" in each cell turns counts into recall per class, which is what you need to see whether one class is being missed more than the other. Here both rows look similar, about 12 and 11 mistakes.

Normalize by rows (actual classes) when you ask "of the real positives, how many did we find?". Normalize by columns (predictions) when you ask "of everything we flagged, how much was right?". State which one you chose in the title.

Step 2: the decision boundary

For a boundary you can see, you need two features, so we switch to a two-dimensional toy problem: make_moons, two interlocking half circles with noise. We compare a logistic regression, which can only draw a straight line, with a random forest, which can bend.

The trick is to evaluate the model on a fine grid covering the plot area. Build the grid with np.meshgrid, flatten it into an (n, 2) array with np.c_, ask the model for predict_proba, and reshape the answers back to the grid's shape. Then contourf fills the probability landscape with color, and a single contour at level 0.5 draws the boundary line. The training points are drawn as circles and the test points as squares, so you can see which ones sit on the wrong side.

Python Decision boundaries of two models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The logistic regression draws one straight line through the moons and cannot do better than around 0.8 here. The forest wraps its boundary around the crescents and reaches nearly perfect training accuracy, but look at its test accuracy: the gain over the straight line is much smaller, and its boundary has small jagged islands around single points. Those islands are the visual signature of overfitting. The colorbar teaches something too: pale bands mean the model is unsure, deep colors mean it is confident.

flowchart LR
    A["meshgrid<br/>of x and y"] --> B["np.c_<br/>(n, 2) array"]
    B --> C["model.predict_proba"]
    C --> D["reshape to<br/>grid shape"]
    D --> E["contourf and contour"]

The recipe for drawing any classifier's decision boundary.

Step 3: ROC and precision-recall curves

A classifier outputs probabilities; "class 1" only appears after you pick a threshold. The ROC curve sweeps that threshold from high to low and plots the true-positive rate against the false-positive rate. The diagonal is a coin flip, and the closer the curve hugs the top-left corner, the better. The area under it (AUC) summarizes the curve in one number.

The precision-recall curve plots the same sweep differently: precision (of what you flagged, how much is right) against recall (of the real positives, how much you found). It is more informative when positives are rare, because ROC can look rosy when negatives vastly outnumber positives.

We compare the forest with a logistic regression on the same test set. roc_curve and precision_recall_curve return the arrays; plotting them is ordinary ax.plot. Setting set_aspect("equal") on the ROC axes makes the diagonal a true 45 degree line.

Python ROC and precision-recall for two models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this dataset the two models are close: the AUCs differ by well under 0.01, and their curves lie almost on top of each other. That is a real and useful finding. A flexible model is not automatically better, and a plot of the whole curve tells you more than one metric: if your application can only tolerate very few false alarms, only the left edge of the ROC curve matters, and there you should compare the curves, not the areas. The dashed "base rate" line on the precision-recall plot is the precision of random guessing, which equals the share of positives (0.5 here, since the classes are balanced).

Step 4: which features matter?

Tree ensembles expose feature_importances_, which measures how much each feature reduced impurity inside the trees. It is free but biased toward features with many distinct values. Permutation importance is an alternative: shuffle one feature's values in the test set and measure how much the score drops. A big drop means the model relied on that feature. Repeating the shuffle ten times gives a standard deviation, which is exactly what error bars are for.

Python Two views of feature importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both charts agree on the winner, feature 5, but they disagree about the rest. The built-in measure gives several features a visible share, while permutation shows that most of them barely affect test accuracy: shuffling them changes almost nothing, and a few bars even dip below zero by chance. The error bars tell you which small bars are indistinguishable from zero. Sort the bars (we sorted by argsort) so the ranking reads from top to bottom, as advised in Bar Charts.

Step 5: the learning curve

Finally, would more data help? A learning curve trains the model on growing slices of the data and records two scores per slice: the training score and a cross-validated score on data the model did not see. Plot both with a shaded band of one standard deviation across the folds (fill_between).

Read it like this. A big gap between the curves means high variance (overfitting): more data or a simpler model should help. If both curves are low and meet, the model is too simple (high bias), and more data will not rescue it. If the validation curve is still rising at the right edge, more data would likely help.

Python Learning curve with confidence bands

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training curve starts at a perfect 1.0 (a handful of examples are easy to memorize) and slowly falls, while the validation curve climbs from about 0.86 to just above 0.91. They are converging but still separated by several points, and the validation line has not fully flattened, so this model would profit from more data or from stronger regularization, such as a smaller max_depth.

Every plot in this lesson was drawn from numbers that scikit-learn returned: arrays of probabilities, counts, rates and scores. Diagnostics are a plotting problem, not a modeling one. Once you have the arrays, everything you learned about subplots, colormaps and annotation applies.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you draw a decision boundary for a two-feature model?

    Matplotlib has no built-in boundary function. You evaluate the model on a dense grid, reshape the output to the grid's shape, fill it with contourf and draw the 0.5 level with contour.
  2. 2. In a confusion matrix drawn with rows as actual classes, what does dividing each cell by its row sum give?

    A row holds all examples of one actual class. The diagonal share of that row is the fraction of that class the model found, which is its recall.
  3. 3. A learning curve shows training accuracy near 1.0 and validation accuracy much lower, with the gap narrowing as data grows. What does that suggest?

    A large train-validation gap is the signature of high variance. If the gap shrinks as the training set grows, more data helps; so can a simpler or more regularized model.
  4. 4. Why can precision-recall curves be more informative than ROC curves for rare positives?

    With few positives, even a small false-positive rate means many false alarms relative to true hits. Precision shows that cost directly, while the ROC curve can still look excellent.

Summary

  • Diagnose a model on held-out data, and plot more than one number: accuracy alone hides the shape of the errors.
  • Decision boundary: meshgrid -> predict_proba -> reshape -> contourf plus contour(levels=[0.5]).
  • Confusion matrix: imshow with counts and row percentages written into the cells.
  • ROC and precision-recall curves come from roc_curve and precision_recall_curve; compare whole curves, not just areas.
  • Permutation importance with error bars is a more honest ranking than built-in impurity importance.
  • A learning curve with fill_between bands shows whether the model needs more data, more regularization or more capacity.

Next project: Physics and Math Visualizer.