Skip to content
Tutorials Center
My learning

Topics

Lesson 100 of 111 ยท Projects

Project: Credit Card Fraud Detection

Build a fraud detector on imbalanced transactions: see the accuracy trap, compare class weights, under-sampling and SMOTE, and choose a threshold by cost.

Fraud detection is the textbook example of imbalanced classification. Out of every 10,000 card payments only a handful are fraudulent, yet each missed fraud costs real money, and each false alarm costs a customer's patience and an analyst's time. A model that says "everything is fine" is almost always right and completely useless.

In this project you will build a detector on synthetic card transactions (we generate them so that the lesson runs anywhere; the techniques are the same on a real dataset such as the public European card-payment data, where the features are anonymous PCA components, like v1, v2 and v3 here). You will:

  1. see the accuracy trap and pick metrics that work for rare events,
  2. compare three ways to handle imbalance: class weights, under-sampling and SMOTE-style over-sampling,
  3. choose the decision threshold by cost, not by habit.

Background lessons: imbalanced classes, classification metrics and thresholds, ROC and precision-recall.

Step 1: The data, the trap and the right metrics

The generator builds 40,000 transactions. Fraud is more likely for large amounts, night hours, online purchases, new merchants, far from home and when many transactions happened in the last day, and it also depends on the anonymous components. About 1.5% of the rows are fraud. We train five models and print many metrics side by side.

Python Step 1: five models, eight numbers each

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom. The model that always answers "legit" has an accuracy of 98.5%, higher than a lot of real models, and finds no fraud at all. Accuracy is therefore useless here. Then notice how differently the other four behave at their default settings:

  • Plain logistic regression is very precise (61.5% of its alarms are real) but finds only 10.8% of the frauds.
  • Logistic regression with balanced class weights finds 90.5% of the frauds, but 9 of 10 alarms are false (precision 0.102), so its F1 is no better.
  • The forest gives the best compromise at the default threshold (F1 0.357).
  • Boosting with default settings misses most frauds, like the plain logistic regression.

Two metrics deserve attention. The Matthews correlation coefficient (MCC) is a single number between -1 and 1 that uses all four cells of the confusion matrix, so it stays honest when classes are very unequal (0 means "no better than chance"). Average precision (AP) summarizes the precision-recall curve and should be compared with the base rate of 0.015, not with 0.5. The AUC values (0.95 and above) look excellent for every model, which is exactly why ROC AUC alone flatters models on rare-event problems. The differences that matter live in precision, recall and AP.

Step 2: Three ways to handle imbalance

We now split the data three ways: train (fit the models), validation (choose between methods and thresholds) and test (touch once at the end). Resampling is only ever applied to the training part; the validation and test rows keep their natural 1.5% fraud rate.

  • Class weights tell the model that an error on a fraud counts more. No data changes.
  • Random under-sampling throws most legitimate rows away so that only ten legitimate rows remain per fraud. It is fast and loses information.
  • SMOTE-style over-sampling (SMOTE: synthetic minority over-sampling technique) invents new frauds on the line between a real fraud and one of its nearest fraud neighbors. It has no library dependency here, because the idea fits in a dozen lines.
Python Step 2: class weights, under-sampling and SMOTE-style over-sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three land close together (average precision 0.34 to 0.36 against a base rate of 0.015, which is about 24 times better than guessing), and the differences between them are within what a different random seed could change. That is a common and useful finding: resampling is not magic. With enough frauds for the model to learn from (356 in the training part), a good algorithm and the right metric, class weights give almost everything the heavier methods give, with less code and no risk of leaking synthetic rows into a validation fold. Resampling matters more when the minority class is tiny, or when the algorithm has no weight option. If you resample inside cross-validation, do it inside each training fold, never before the split (nested CV and leakage).

Step 3: Choose the threshold by cost

The default threshold of 0.5 is a statistical habit, not a business decision. What the bank cares about is money:

  • A missed fraud loses the transaction amount.
  • Every flagged transaction costs $3 of review time, whether it turns out to be fraud or not.

So the best threshold minimizes lost money + review cost. We find it on the validation set and then evaluate it once on the test set, next to three simple policies.

Python Step 3: pick the threshold that minimizes expected cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Flagging nothing loses $17,249 to fraud in the test set. The default threshold of 0.5 catches only 38 of the 119 frauds and costs $9,160. The threshold chosen by cost is very low (0.10): it raises 793 alarms but catches 97 of the 119 frauds and cuts the total cost to $3,920, a reduction of about 77% against doing nothing. Even a handmade rule such as "flag amounts above $150" beats the default threshold, which tells you how much the default can cost. Notice the price of this policy, though: most alarms are false (about 700 of 793), and the review team must be able to handle that volume. If reviewers are scarce, change the cost parameters or cap the number of alarms per day and keep the highest-scoring ones.

What to do on real data

  • Keep the split honest: fraud changes over time, so split by time (train on older months, test on the latest) rather than at random (cross-validation for special data).
  • Treat features carefully: anything recorded after the transaction, such as a later chargeback flag, is leakage.
  • Calibrate probabilities if you will use them in cost formulas (calibration).
  • Combine the model with rules and an anomaly detector for new attack patterns that the labeled history has not seen.
  • Monitor precision, recall and the input distributions after launch, because fraudsters adapt (monitoring).
  • Explain flagged transactions to analysts (feature importance).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. On data with 1.5% fraud, a model that always predicts 'legit' has about what accuracy?

    It is right on every legitimate row, which is 98.5% of them, and wrong on every fraud. That is why accuracy misleads on rare events.
  2. 2. Which statement about resampling is correct?

    Synthetic or duplicated rows in a validation or test set would make the scores unrealistic, and rows copied across the split leak information.
  3. 3. Why did the cost-based threshold (0.10) beat the default threshold of 0.5 in the project?

    The costs were asymmetric, so the best operating point favors recall. With other costs the best threshold would change.
  4. 4. What does the Matthews correlation coefficient add compared with accuracy?

    MCC is a correlation between predictions and labels, so the always-legit model scores 0 although its accuracy is 98.5%.

Summary

  • On rare-event problems accuracy and ROC AUC flatter bad models; use precision, recall, F1, MCC and average precision, and compare AP with the base rate.
  • Class weights, under-sampling and SMOTE-style over-sampling gave similar results here; apply resampling only to training data.
  • Split into train, validation and test, choose methods and thresholds on validation data, and report the test result once.
  • Choose the threshold from the real costs of missed frauds and false alarms, and make sure the team can handle the alarm volume.

Next: another prediction problem that is famous for tempting mistakes: Stock and Crypto Price Prediction.