Skip to content
Tutorials Center
My learning

Topics

Lesson 98 of 111 ยท Projects

Project: Credit Risk Scoring

Build a credit default model end to end: audit for target leakage, add monotonic constraints to gradient boosting, calibrate probabilities, and check fairness by group.

Credit scoring is where machine learning meets rules that matter. A model decides who gets a loan and at what price, so a leaked feature, an uncalibrated probability or an unexamined bias is not a small bug: it becomes a bad business decision or an unfair one. That is why this project spends less time on the algorithm than on the checks around it.

You will work on a synthetic loan book with a hidden default process. The steps follow the order a careful team would use: audit the columns for leakage before modelling, train a gradient boosting model with monotonic constraints so its behavior matches common sense, calibrate its probabilities so that "20%" really means about one in five, and finally check how the decisions fall across groups. Everything is generated with fixed seeds, so the numbers in the text match your runs.

Four checks around the model

Leakage audit Would this column exist at decision time?
Monotonic rules More debt must never mean less risk.
Calibration Predicted risk should match real rates.
Fairness check Compare decisions across groups.

The model is the easy part. The checks are the project.

1 / 2

Step 1: The data and a leakage audit

The generator builds 6,000 loans. Default depends on the debt ratio, late payments in the last twelve months, credit utilization, income and length of credit history. About 8% of incomes are missing, as in a real application form. The table also contains a column called collections_calls, the number of calls the collections department made. It is tempting, because it is in the same table, but think about when it is filled in: collections only call people who have already stopped paying.

That makes it a case of target leakage (Nested Cross-Validation and Data Leakage). At the moment a loan is approved, collections_calls does not exist yet. A model that uses it will look wonderful on paper and be useless in production. The audit below measures how well each column alone separates defaulters from non-defaulters (the AUC, where 0.5 is useless). A single column that scores far above the others is a red flag, not a prize.

We also gather gender and region here. They will not be used as model inputs, but we keep them in the table for the fairness check in step 3.

Python Step 1: generate the loan book and audit the columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The honest features each reach an AUC of only about 0.56 to 0.60 alone, which is normal for credit data, where no single column is decisive. collections_calls reaches about 0.97 on its own, which is not plausible for a real underwriting signal, and that is the clue. The second table settles it: removing that one column drops the model's test AUC from about 0.97 to about 0.65. The high score was the leak, and the honest score is the real one.

When a model on messy real-world data gets a score that looks too good, treat it as a bug until proven otherwise. Ask for every column: when is this recorded, by whom, and could it be a consequence of the outcome?

Step 2: Monotonic constraints

An unconstrained boosted model fits whatever pattern the training data suggests, including noise. In credit scoring that can produce nonsense that is easy to spot and hard to defend: a model that gives a lower risk to an applicant with a higher debt ratio, just because a few unusual training rows pointed that way. Regulators, risk officers and applicants all expect common-sense monotonic behavior: risk should not fall when debt, late payments or utilization go up, and should not rise when income or credit history grows.

HistGradientBoostingClassifier supports this directly through monotonic_cst (Histogram Gradient Boosting). You pass +1 for "risk must not decrease as this grows", -1 for "must not increase", and 0 for no constraint. The model also handles the missing incomes natively. We compare both models on the test set and then trace the risk of a typical applicant while only the debt ratio changes.

Python Step 2: unconstrained vs monotonic gradient boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The constrained model is not worse. Its test AUC is about 0.66 against about 0.65 for the free model, and the Brier scores are essentially the same, so the rules cost nothing here, and on noisy data they often act as a mild regularizer. The sweep shows the difference in behavior. The unconstrained model zig-zags: its predicted risk for the typical applicant goes up, then down, then up again as debt grows. The constrained model climbs and then stays flat; it never decreases. That is a model you can explain to a risk committee.

Notice also that the AUC is modest. A model that looks only at honest signals is meant to be modest: credit outcomes contain a lot of luck, so a good scorecard ranks borrowers a bit better than chance rather than perfectly.

Step 3: Calibrate and check fairness

A score used for pricing has to be a probability. If the model says 20%, about one in five such loans should default, otherwise the interest rates built on it are wrong (Probability Calibration and Uncertainty). Boosted trees often produce probabilities that are a little too confident or too compressed, so we wrap the model in CalibratedClassifierCV with isotonic regression and compare Brier scores and a reliability table.

Then comes the fairness check (Fairness, Bias and Responsible ML). The model never saw gender or region, but leaving a column out does not make a model fair: other columns can act as proxies. In our generator, region is correlated with income. We apply a decision rule (decline if the predicted risk is at least 25%) and compare, for each group, how many applicants are declined, the actual default rate, how many good customers are wrongly declined, and how many bad ones are caught.

Python Step 3: calibration and a per-group audit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Calibration helps a little. The Brier score improves from about 0.1439 to about 0.1397, and the mean predicted risk moves to 0.179, matching the actual default rate of 0.179. In the reliability table the predicted and actual rates track each other across the five risk bins, with small deviations in the middle bins, which is what a calibrated model on 1,800 test rows looks like.

The fairness table needs careful reading. By gender, the decline rates are close (about 20% and 22%), and the small difference follows the small difference in actual default rates. By region the picture is more uneven: the south region has the highest decline rate (about 26%) and also the highest share of good customers wrongly declined (about 22%, against 14% in the west). The south has lower incomes in this data, and income is a model input, so the model transmits that difference. Whether this is acceptable is not a statistical question. It depends on the law in your country, the business justification for each feature, and which fairness definition your organization adopts, because several common definitions cannot all hold at once when base rates differ.

With only a few hundred applicants per group, differences of a percentage point or two are within the noise. Report group sizes next to the rates, and compute confidence intervals before acting on small gaps.

What a project like this should deliver is not only a model. It is a short model card: which columns were used and why, the leakage audit, the constraints and their reasons, calibration results, the fairness table with its caveats, and a list of conditions under which the model should be retrained or retired.

Where to go next

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is collections_calls a leaky feature for predicting default at application time?

    Collections activity is a consequence of default. A model trained on it learns the outcome itself, scores brilliantly in testing, and has nothing to work with in production.
  2. 2. What does monotonic_cst={'debt_ratio': 1} promise?

    A +1 constraint forces the model's output to be non-decreasing in that feature. It encodes domain knowledge and removes implausible zig-zags, often at little or no cost in accuracy.
  3. 3. A model never receives the region column, yet its decline rate differs by region. How is that possible?

    Removing a sensitive column does not remove its influence if other features carry the same information. That is why outcomes must be checked by group, not just inputs.
  4. 4. What does calibration check that AUC does not?

    AUC measures ranking only. A model can rank perfectly while its probabilities are all too high or too low, which breaks any pricing or expected-loss calculation built on them.

Summary

  • Audit every column for leakage by asking when it is recorded; a single feature with an implausibly high score is the usual symptom.
  • Honest credit models are modest in AUC, and removing the leaky column in our data cut the test AUC from about 0.97 to about 0.65.
  • Monotonic constraints in HistGradientBoostingClassifier encode domain knowledge, remove zig-zag behavior and here cost no accuracy.
  • Calibrate scores that will be used as probabilities and check them with the Brier score and a reliability table.
  • Leaving sensitive columns out does not make a model fair; compare decisions and error rates by group, report group sizes, and document the result.

Next, test your understanding of the whole course so far in Exercises: Workflow, Regression and Classification.