Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 39 ยท Projects and Practice

Project: Fraud Detection

Detect rare fraudulent transactions: simulate imbalanced data, see why accuracy fails, train supervised and unsupervised detectors, tune the threshold by cost and report precision and recall.

A payment company loses money to fraud, but fraud is rare: about 1 transaction in 200. Analysts can only review a limited number of alerts per day, so the model must rank transactions so that the top of the list is rich in real fraud. This project combines classification, ensembles and anomaly detection, and shows how to handle imbalanced classes.

The data is simulated so you can run it anywhere. Each block is self-contained, so the generator is repeated.

Step 1: The data and the baseline

Features: amount, hour of day, distance from the home location (km), number of transactions in the last hour and whether the merchant is new to the customer. Fraudulent transactions tend to be larger, at night, far from home and in bursts, but with plenty of overlap, as in life.

Python Simulate and measure the naive baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline is 99.5% accurate and catches no fraud, so accuracy is useless here. We will use precision, recall and average precision.

Step 2: A supervised model

Train on part of the data and test on the rest, stratified so both parts contain frauds. Random forests handle the mix of features well; class_weight="balanced_subsample" makes the rare class count more.

Python Random forest and ranking metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The default 0.5 threshold is arbitrary. With rare events the best threshold is almost always lower or higher than 0.5, depending on what each error costs.

Step 3: Pick the threshold by cost

Suppose a missed fraud costs 200 on average, and every alert that analysts review costs 5 (their time). The cost of a threshold is 200 * missed frauds + 5 * flagged transactions. Search for the threshold with the lowest cost on validation data.

Python Cost-based threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The threshold is chosen on a validation set and its cost is reported on a separate test set; choosing and reporting on the same data would be optimistic.

Step 4: No labels? Unsupervised detection

In many real systems confirmed fraud labels are few and late. An unsupervised detector such as Isolation Forest scores how unusual each transaction is without labels, then labels are used only to check the ranking.

Python Isolation Forest versus the supervised model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Supervised learning wins when good labels exist; the unsupervised detector still finds many frauds with no labels at all and can complement it by catching new fraud patterns the labeled data has never shown.

Step 5: Deployment notes

  • Drift: fraudsters adapt. Monitor precision and the share of flagged transactions weekly, and retrain regularly.
  • Leakage: features such as "chargeback received" are known only after the event; do not use them for training.
  • Time-based splits: in production test on later data than you train on.
  • Explanations: analysts trust alerts more with reasons ("amount 8x the customer's average, new country"). Tree-based feature importances and per-alert comparisons help.
  • Fairness: audit that the model does not unfairly flag groups of customers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is accuracy a bad metric for fraud detection with 0.5% fraud?

    With rare events the majority class dominates accuracy, so use precision, recall and average precision.
  2. 2. What does class_weight='balanced_subsample' do in a random forest?

    Errors on the rare class count more during training, which counters the imbalance.
  3. 3. Why choose the decision threshold on validation data and report cost on test data?

    A threshold tuned on the data it is scored on looks better than it will in practice.
  4. 4. When is Isolation Forest especially useful for fraud?

    It scores unusualness without labels and can catch new, unseen patterns.

Summary

  • With rare events judge models by precision, recall and average precision, not accuracy.
  • Handle imbalance with stratified splits and class weights; pick the threshold from the real costs of misses and false alarms using validation data.
  • Unsupervised detectors complement supervised models when labels are scarce.
  • Plan for drift, leakage and explanations before deploying.

Next: Project: Spam Filter and Topic Explorer