Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 40 ยท Projects and Practice

Project: Fraud Detection

Build a complete fraud model on generated transactions: time-based split, imbalance, PR-AUC, a cost-based threshold, top-k review lists and per-alert explanations with SHAP contributions.

In this project you assemble the pieces of the last chapters into one workflow on a realistic problem: flagging fraudulent card transactions. Fraud is rare, costly to miss, expensive to review, and changes over time, which makes it a good test of everything you know about boosted trees. Real card data is private, so the project generates a synthetic table with the same difficulties. No data is downloaded.

The steps are: build the data, split by time, train and evaluate on PR-AUC, choose a threshold from costs, produce a review list, and explain individual alerts.

flowchart LR
  A["Transactions<br/>(1.5% fraud)"] --> B["Time-based split"]
  B --> C["Train XGBoost<br/>early stopping, PR-AUC"]
  C --> D["Threshold from costs<br/>chosen on validation"]
  D --> E["Test: alerts, cost,<br/>top-k review list"]
  E --> F["Explain each alert<br/>with SHAP contributions"]

The workflow of the project. The test set is touched once, at the end.

Step 1: generate the data

Each transaction has an amount, an hour of the day, a merchant category, the card's number of transactions in the last 24 hours, whether the country differs from the card's home country, and the account age. Fraud is more likely for large amounts at night, in unfamiliar countries, on brand-new accounts and in bursts of activity. The rows are in time order, which matters for the split.

Python Step 1: build and inspect the transaction table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the fraud rate was 2.1% (256 frauds in 12,000 rows), foreign transactions had a fraud rate of 8.6% against 1.6% for domestic ones, and night-time ones 5.9% against 1.1%, as designed. The table is saved to tx.csv in the runner's temporary file system; the next steps regenerate it with the same seed so that each block stands alone.

Step 2: time split, model, PR-AUC

We keep the earliest 60% of the rows for training, the next 20% for validation (early stopping and the threshold) and the last 20% for the final test. This mimics deployment: train on the past, evaluate on the future.

Python Step 2: train with early stopping on PR-AUC

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the validation PR-AUC was 0.255 and the test PR-AUC 0.257, about ten times the random baseline of roughly 0.02 (the fraud rate), with ROC-AUC values of 0.86 and 0.84. That is far above random but far from 1.0, which is normal for a rare, noisy event. Early stopping on aucpr picked a best iteration of only 8, because the metric is noisy with 61 validation frauds; a larger patience or a smoother stopping metric would grow more trees, which is a good experiment to try. With only 50 to 60 frauds in each part, expect scores to move by several hundredths between splits. The importance table put account_age_days, hour, foreign and amount on top, as built into the data, and the weak merchant_cat last. If importance disagreed with how you built the data, you would have found a bug.

Step 3: threshold, cost and review list

Suppose a missed fraud costs 200 on average, and reviewing an alert (a false alarm) costs 5. Choose the threshold on the validation predictions, then apply it once to the test set. Also build the alternative a team with limited reviewers would use: a top-k list.

Python Step 3: cost-based threshold and top-k list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the chosen threshold was 0.09. On the test set the model flagged 483 of 2,400 transactions and caught 40 of the 50 frauds, for a total cost of 4,215, against 10,000 for flagging nothing and also 10,000 for the default threshold 0.5 (the model never reaches 0.5 here, so it flags nothing). Flagging nothing is the baseline that a business already has, and the cost-optimal threshold beats it by more than half. With a cost of 200 against 5, the best threshold is very low, because reviewing 40 alerts costs the same as missing one fraud. The top-2% list (48 transactions, 14 of them frauds, a precision of 29%) is the alternative when analysts can only handle a fixed number of cases: it needs no threshold, and its precision tells you how productive each review is.

Step 4: explain an alert

Analysts trust an alert more when they see why it fired. XGBoost returns SHAP contributions in log-odds with pred_contribs=True (SHAP Values for Tree Models). The code below prints the top reasons for the highest-scoring test transaction:

Python Step 4: why was this transaction flagged?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the highest-risk transaction had a large amount (526.61, contribution +0.68 log-odds), a new account (30 days, +0.37) and a transaction at 4 a.m. (+0.25), which is exactly what you would show an analyst as "large amount, new account, night-time". The last line confirms additivity: the base value plus the contributions equals the model's raw margin (-1.04). The contributions are computed with the same trees as predict by passing iteration_range; without it, the raw booster would use all trees that were trained after the best iteration, and the numbers would not match. In a real system you would store these top reasons next to each alert, and use them to debug the model: a reason that makes no business sense is often a data problem.

Extensions

  • Replace XGBoost with LightGBM and compare PR-AUC on the same split.
  • Add engineered features such as the ratio of the amount to the card's average amount, and the time since the last transaction.
  • Train with scale_pos_weight and watch what happens to the threshold and the probabilities (Class Imbalance).
  • Simulate drift: change the generator after the split (more night-time fraud) and see how the test PR-AUC falls (Pitfalls).
  • Wrap the steps in a function that saves the model, its feature list and the threshold (Saving and Loading Models).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why was the data split by time instead of randomly?

    Fraud patterns change over time; a time split gives an honest estimate of future performance.
  2. 2. On which data was the decision threshold chosen?

    Choosing on validation data keeps the test result honest.
  3. 3. A missed fraud costs 200 and a review costs 5. What should happen to the best threshold compared with 0.5?

    Misses are far more expensive than reviews, so the cost-optimal rule flags many more transactions, at lower scores.
  4. 4. What is the benefit of a top-k review list?

    The team reviews the k highest scores regardless of the score scale, which also stays usable when the distribution drifts.

Summary

  • A good fraud workflow uses a time-based split, PR-AUC for stopping and evaluation, and a threshold chosen from business costs on validation data.
  • Compare against the baselines (flag nothing, default 0.5) to prove the model has value.
  • Top-k lists suit fixed review capacity; SHAP contributions give analysts reasons for each alert.
  • Check importances against how you built the data, and expect small test sets to give noisy scores.

Next: Project: House Price Regression