Skip to content
Tutorials Center
My learning

Topics

Lesson 104 of 111 ยท Projects

Project: Heart Disease and Breast Cancer Diagnosis

Two diagnosis projects with scikit-learn: breast cancer cell measurements and a heart-disease table. Compare logistic regression and kNN with cross-validation and tune for recall.

In most machine learning problems the two kinds of mistake cost about the same. In medicine they do not. Telling a healthy person "we want a second look" costs an afternoon and some worry. Telling a sick person "all clear" can cost a life. A diagnosis model therefore has to be judged on recall (the share of sick patients it finds) first, and on precision (how many alarms are real) second.

This project builds two screening models. The first uses the breast cancer data set that ships with scikit-learn: 569 tumors, each described by 30 measurements of cell nuclei from a biopsy image, labeled malignant or benign. The second is a heart-disease table with the usual kinds of columns (age, sex, chest pain type, blood pressure, cholesterol, heart rate, exercise symptoms). Real hospital records need consent and a download, so the code generates this one with a fixed seed from a known risk formula. Its numbers say nothing about real patients. The method (careful cross-validation, recall-focused thresholds, readable models) carries over to real data.

These are learning exercises, not medical advice or a medical device. A real diagnostic model needs clinical validation, regulatory approval and monitoring, and it supports a doctor's decision rather than replacing it. See Fairness, Bias and Responsible ML.

What matters in diagnosis

Recall first Find the sick patients.
Cost of a miss A missed case is the worst error.
Honest testing Cross-validate, test once.
Explainable Doctors need reasons.

Accuracy alone hides the errors that matter.

1 / 2

Project 1: Breast cancer

Step 1: Compare models with cross-validation

Scikit-learn codes the target as 0 for malignant and 1 for benign, which is the opposite of what a screening metric wants. Our first line flips it so that malignant is the positive class (1). Then "recall" means "the share of cancers found".

We hold out 25% of the rows as a test set and use 5-fold stratified cross-validation on the rest to compare a majority baseline, k-nearest neighbors with and without scaling, and logistic regression. The two distance-based and gradient-based models need scaling, so the scaler sits inside a pipeline, fitted on training folds only. A final grid search picks k for kNN by cross-validated recall.

Python Breast cancer: baseline, kNN, logistic regression and tuning k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline shows the data is a little lopsided (37% malignant): predicting benign every time gets 62.7% accuracy and finds no cancer at all. The numbers also show why scaling matters. The 30 measurements have very different scales (an area is in the hundreds, a smoothness value near 0.1), and an unscaled kNN is dominated by the big ones: recall 0.881 against 0.931 once the features are standardized. Logistic regression is the best model on every metric: 97.7% accuracy, recall 0.962 and an AUC of 0.993. The k search is also useful for what it does not find. Recall is highest for k = 1 (0.943) and falls to 0.875 for k = 15, so smaller neighborhoods are better here. Even the best kNN is still below logistic regression, so we continue with logistic regression.

Step 2: Choose the threshold for the cost of a miss

At the default threshold of 0.5, the logistic regression misses about 4% of the cancers in cross-validation. With these stakes, we want to move the threshold down: flag a patient whenever the predicted probability of malignancy is above, say, 0.1. That catches more cancers and causes more false alarms (Thresholds, ROC and Precision-Recall Curves).

The important detail is where the threshold is chosen. The code gets out-of-fold probabilities for the training part with cross_val_predict, lists recall and precision for several thresholds, and applies a simple rule: the highest threshold that still reaches 98% recall. Only then does it fit on the whole training part and look at the test set, once.

Python Choose a recall-focused threshold on training data, test once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table shows the trade. Going from 0.5 to 0.05 lowers the cross-validated misses from 6 to 2, and false alarms rise from 4 to 43. The rule picks 0.09. On the untouched test set, the default threshold finds 51 of 53 cancers and raises no false alarm, while the chosen one finds all 53 and raises 12 false alarms (precision 0.815). Whether 12 extra follow-up visits are worth two found cancers is a medical and ethical decision, not a statistical one, but now the numbers are on the table. The model's ROC AUC of 0.996 on 143 test patients also needs a caveat: with 53 malignant cases, one extra miss changes recall by about 2 points, so the real uncertainty is wider than the printed digits.

The strongest weights are measurements of the size and shape of cell nuclei (worst radius, worst concave points), which agrees with what pathologists look at. Checking that a model relies on sensible inputs is part of feature importance work, and it is what a doctor will ask first.

Project 2: Heart disease

Step 3: A harder, messier table

The heart-disease table has numeric columns (age, blood pressure, cholesterol, maximum heart rate, ST depression), categorical columns (sex, chest pain type, exercise-induced angina) and 35 missing cholesterol values. A ColumnTransformer handles each type: median imputation and scaling for the numbers (Handling Missing Data), one-hot encoding for the categories (Categorical Encoding). Everything lives in one pipeline, so every fold fits its own imputer, scaler and encoder.

A table of 700 rows is small, so one 5-fold run is noisy. We use repeated stratified cross-validation (5 folds, 3 times) and report the mean and the spread over 15 folds. We compare four models: the baseline, logistic regression, logistic regression with class_weight="balanced" (which penalizes missed positives more), and kNN.

Python Heart disease: pipeline, repeated CV and class weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This problem is harder than the cancer one, and the output says so. About 26% of patients have the disease. Logistic regression's AUC is 0.78, and at the default threshold its recall is only 0.407: it finds fewer than half of the sick patients, with precision 0.674. The model separates the groups, but its 0.5 cut is too cautious for a screening goal, because only 26% of patients are positive. Two ways to move toward recall appear in the table. Setting class_weight="balanced" raises recall to 0.677 and drops precision to 0.458, the same trade-off as the lowered threshold in the cancer project. kNN is worse on every measure (AUC 0.70, recall 0.237), because the class overlap is large and a local average is noisy. The +/- values (about 0.06 for recall) are the reason to use repeated CV: a difference smaller than that between two models is not evidence of anything.

Because logistic regression is readable, the last lines show its odds ratios: how much the odds of disease multiply for each unit of a feature (numbers are standardized, so a unit is one standard deviation). No chest pain symptoms (chest_pain_none) multiplies the odds by about 3, ST depression by 1.8 per standard deviation and exercise angina by 1.6. These are the effects we put into the generator, so we know the model recovered the right signs. With real data the same table is what you would discuss with a clinician. Compare it with what is known from medicine, and investigate every surprise.

Lessons from both projects

  • Choose the metric from the cost of errors. For screening, recall first, then precision at that recall.
  • Never tune on the test set. Choose the threshold, k and class weights with cross-validation on training data, then evaluate once.
  • Scale for distance and gradient models, inside the pipeline.
  • Report uncertainty. Small data sets give wide error bars; repeated CV and the test-set size tell you how wide.
  • Prefer a readable model that is nearly as good. Logistic regression won the cancer task and gives odds ratios for the heart task.
  • Look at who the data represents. A model trained on one hospital's patients may not work elsewhere (Fairness, Bias and Responsible ML, monitoring).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is recall usually the first metric for a cancer-screening model?

    Recall is the share of real cases found. When missing a sick patient is the worst outcome, you want to maximize recall and then limit the false alarms.
  2. 2. Scaling improves kNN on the breast cancer data. Why?

    Area is measured in hundreds and smoothness near 0.1. Without scaling, the distance is determined almost entirely by the large-valued features.
  3. 3. Where should you choose a recall-focused threshold?

    If the threshold is chosen on the test set, the reported test performance is optimistic. Out-of-fold predictions on the training data are a safe place to choose it.
  4. 4. On the heart-disease table, class_weight='balanced' raised recall from 0.41 to 0.68. What is the price?

    Weighting the positive class more makes the model flag more patients. Recall rises, but a larger share of the flagged patients are healthy, so precision drops (0.67 to 0.46 here). The AUC does not change because the ranking is almost the same.

Summary

  • Diagnosis models are judged on recall first, precision second, and the threshold encodes the cost of a miss.
  • Flip the target so the disease is the positive class, and build pipelines (imputer, scaler, encoder, model) so cross-validation is honest.
  • Scale features for kNN and logistic regression; compare models with cross-validation, repeated when the data is small.
  • Choose thresholds and class weights on training data, report test results once, and mention the uncertainty of small test sets.
  • Logistic regression gives readable odds ratios; check them against domain knowledge.

Next: recommend movies in Project: Movie Recommender.