Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 73 ยท Projects

Project: Chest X-ray Pneumonia and Lung Cancer Detection

Build a medical-imaging classifier project: class imbalance, recall versus precision, threshold choice and occlusion maps, plus the transfer-learning recipe.

Medical imaging is one of the most meaningful uses of deep learning, and one of the places where careless habits do the most harm. A model that reads a chest X-ray and says "pneumonia" or "no pneumonia", or that looks at a CT scan for lung nodules, is easy to build and hard to build well. The hard parts are not the network. They are the imbalance (most scans are healthy), the cost of mistakes (a missed cancer is far worse than a false alarm), and the many ways a model can score well on paper while learning the wrong thing.

This project teaches those parts. As in the other image projects, the browser cannot download hospital data or train on a GPU, so the experiments use synthetic chest-like images: dark lung fields on a brighter body, with a faint bright opacity inside a lung in the positive cases. Every number comes from code that runs here. The real-data recipe follows as read-only PyTorch. Nothing in this lesson is medical advice, and a model trained on a toy or on public data is a learning exercise, not a diagnostic tool.

Why medical imaging is different

Imbalance Most scans are healthy, so accuracy lies.
Unequal costs A missed disease is worse than a false alarm.
Hidden shortcuts Scanner, hospital and markers can leak the label.
1 / 2

The real data sets

Two public collections are the usual starting points. The Chest X-Ray Images (Pneumonia) data set, popular on Kaggle, holds about 5,900 pediatric chest X-rays labeled normal or pneumonia, with roughly three times as many pneumonia images as normal ones in its training split. The NIH ChestX-ray14 data set has over 100,000 frontal X-rays, each labeled with any of 14 findings. For lung cancer, the usual material is CT scans: the LUNA16 challenge (a curated subset of the LIDC-IDRI collection) provides scans with annotated lung nodules, and the 2017 Data Science Bowl asked whether a patient would develop cancer. CT scans are 3D volumes, and nodules can be a few millimeters across inside hundreds of slices, so lung-cancer pipelines typically first detect candidate nodules and then classify small 3D crops.

Always read each data set's license and documentation, and understand how the labels were produced (a radiologist's reading, a text-mined report, a later biopsy). Label quality sets the ceiling on what any model can achieve.

Step 1: the accuracy trap and class weights

In our synthetic screening set, about 10% of the images show an opacity. Look at what that does to the most common metric: a "model" that says healthy for every patient scores about 90% accuracy and finds none of the sick patients. Accuracy alone is therefore meaningless here. The metrics that matter are:

  • Recall (also called sensitivity): the fraction of sick patients the model finds, TP / (TP + FN).
  • Precision: the fraction of alarms that are real, TP / (TP + FP).
  • Specificity: the fraction of healthy patients correctly cleared, TN / (TN + FP).
  • F1: the harmonic mean of precision and recall, a single number that is low if either is low.

See Evaluation Metrics for the definitions in depth. The network is the small CNN from Training a CNN From Scratch, with one change: instead of flattening the pooled map, it takes the global maximum of each filter's response over the whole image. A filter then acts as a detector, "is there a bright blob anywhere", which suits an opacity that can appear in any position. We train it twice, with the ordinary loss and with a class-weighted loss that makes each positive example count about nine times as much (the ratio of healthy to sick images), then compare the models on a held-out test set.

Python Imbalanced data: plain loss versus class weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "always healthy" baseline scores 89.8% accuracy and finds 0 of 61 patients. The plain-loss network scores 98.8% accuracy with a recall of 91.8%: it misses 5 of 61 sick patients and raises only 2 false alarms. With the class-weighted loss, accuracy drops to 96.7%, but recall rises to 98.4% (1 patient missed) and false alarms rise to 19, so precision falls from 96.6% to 75.9%. The weighted model is worse on accuracy and better on what matters here, because it is willing to raise more alarms to avoid missing a patient. Which model is right depends on the cost of each kind of mistake, which is a clinical question, not a statistical one. In this toy the signal is strong, so even the plain loss does well. On real X-rays, with subtle findings and 1% prevalence, the gap between plain and weighted training is typically much larger.

Other imbalance remedies are oversampling the rare class (or undersampling the common one), focal loss (down-weights easy examples so rare, hard ones dominate), more careful augmentation of the positive class, and simply collecting more positive cases. Class weights are the quickest to try.

Step 2: choose the operating point

A classifier outputs a probability, and the 0.5 cut-off is only a default. Moving the threshold trades one error for the other without retraining. The next program sweeps the threshold on the weighted model's test predictions, and then looks at what the model is attending to with occlusion sensitivity: slide a gray square across one positive image, and record how much the "pneumonia" probability drops each time. If the model is looking at the right place, covering the opacity should hurt the score and covering anything else should not. This is a simple cousin of Grad-CAM, the standard heat-map method for CNNs, which computes the same kind of map from gradients in one pass.

Python Threshold sweep and an occlusion map

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The threshold table shows the whole trade. At 0.05 the model finds every patient but raises 337 false alarms (precision 15.3%); at 0.5 it finds 98.4% with 19 false alarms; at 0.95 there are no false alarms at all but it misses 27 of the 61 patients (recall 55.7%). For a screening test you would pick a point near the top of the table (high recall) and accept follow-up scans for the false alarms. A confirmation test would choose the opposite. In practice, you choose the threshold on a validation set, never on the test set, and report the result on the test set.

The occlusion map confirms that the network found the right evidence: for patient 6 the opacity sits near row 8, column 9, and the bright cells in the map cluster around the lower right of the image, where hiding the patch reduces the probability most. If the map had lit up a corner marker or the image border instead, you would know the model had found a shortcut.

Step 3: the real recipe

On real X-rays the pieces are the ones from Transfer Learning and Fine-Tuning: a pretrained backbone, a new head, and a loss that accounts for imbalance. This code is read-only.

Python
# Read-only: chest X-ray classification with a pretrained DenseNet-121 (needs PyTorch, torchvision, a GPU)
import torch, torch.nn as nn
from torchvision import models, transforms

tf = transforms.Compose([transforms.Grayscale(num_output_channels=3),      # X-rays are 1 channel; the backbone expects 3
                         transforms.Resize((224, 224)),
                         transforms.RandomRotation(7),                       # mild augmentation only: anatomy is not symmetric
                         transforms.ColorJitter(brightness=0.2, contrast=0.2),
                         transforms.ToTensor(), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

model = models.densenet121(weights=models.DenseNet121_Weights.DEFAULT)
model.classifier = nn.Linear(model.classifier.in_features, 1)              # one logit: pneumonia or not

n_neg, n_pos = 1341, 3875                                                  # counts from YOUR training split
loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(n_neg / n_pos))     # weight the rarer class up

# Evaluate with recall, precision and ROC AUC on a validation set; choose the threshold there.
from sklearn.metrics import roc_auc_score, precision_recall_curve

Notes on that recipe: use a mild augmentation (small rotations, brightness and contrast changes); horizontal flips put the heart on the wrong side and rarely reflect real scans. Use the pos_weight of BCEWithLogitsLoss for the imbalance. Train the head first with the backbone frozen, then unfreeze the top blocks with a small learning rate. Evaluate with ROC AUC, the precision-recall curve and recall at a chosen specificity, not accuracy.

What goes wrong in real medical projects

  1. Patient leakage. One patient may have many images. If images of the same person land in both the training and the test set, the model can recognize the patient rather than the disease, and the test score is inflated. Split by patient, never by image.
  2. Shortcuts. Models learn whatever predicts the label, including hospital tags, portable-scanner markers, patient position, or image brightness. Public pneumonia data sets have documented cases of this. Check explanation maps and test on images from a different hospital (external validation).
  3. Distribution shift. A model trained on one scanner or population can fail on another. Report performance by subgroup (age, sex, device).
  4. Label noise. Radiologists disagree on subtle findings, and labels mined from reports are noisy. Know your labels.
  5. Calibration. A probability of 0.8 should mean about 80% of such cases are positive. Check with a calibration curve before showing probabilities to anyone.
  6. Deployment and regulation. A clinical tool is a regulated medical device, with its own testing, monitoring and documentation requirements. See Ethics, Bias and Safety.

Ideas to extend the project

  • Train with focal loss and compare its precision-recall curve with class weights.
  • Compute the ROC and precision-recall curves for the NumPy model by sweeping the threshold, and report the area under each.
  • Make the opacities fainter and fewer, and watch how plain loss degrades faster than weighted loss.
  • Move from classification to localization: predict a box or a mask for the opacity, as in Object Detection and Image Segmentation.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a screening set where 10% of patients are sick, a model that always predicts healthy gets about 90% accuracy. What does this show?

    The trivial model is useless to patients (it finds nobody), yet it scores well on accuracy. Recall, precision and F1 reveal the failure.
  2. 2. Lowering the decision threshold from 0.5 to 0.05 in the experiment did what?

    A lower threshold labels more cases positive: every patient was found, but 337 healthy images were flagged too.
  3. 3. Why should you split medical images by patient rather than by image?

    If the same person appears in both splits, the model may recognize the person, not the disease. A patient-level split gives an honest estimate.
  4. 4. What does an occlusion map (or Grad-CAM) tell you?

    It shows which regions change the output most. If it highlights a marker or border instead of the lesion, the model has learned a shortcut.
  5. 5. How does a class-weighted loss change the trade-off between recall and precision?

    Weighting positives more makes the model prefer raising an alarm to missing a case, so it finds more patients at the price of more false alarms.

Summary

  • With rare positives, accuracy is misleading; report recall, precision, specificity, F1 and ROC or precision-recall curves.
  • Class weights, oversampling or focal loss make the rare class count; they typically trade precision for recall, and the right balance depends on the cost of each error.
  • The decision threshold is a deliberate choice made on a validation set, not a fixed 0.5.
  • Global max pooling suits detecting a lesion at an unknown position, and occlusion maps or Grad-CAM show whether the model looks at the right evidence.
  • Real projects start from a pretrained backbone, use mild augmentation (no unrealistic flips), and a pos_weight loss.
  • Split by patient, check for shortcuts, validate externally, check calibration and respect regulation. A toy or public-data model is never a diagnostic tool.

Next project: Text Generation with an LSTM.