Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 73 ยท Frameworks and Production

Responsible Deep Learning

Build models that are fair, private and robust: measure bias with fairness metrics, try a gradient attack, read saliency maps and write a model card.

A model that scores well on a test set can still do harm. It might work much worse for one group of people than another, leak details of its training data, be fooled by a tiny change that a human would never notice, or be trusted in a situation it was never designed for. These are not exotic problems; they come from the same ingredients as everything else in this course: data, a loss function and an optimizer that does exactly what it is told.

Responsible deep learning is the habit of asking "who could this fail, and how would I know?" before shipping. This lesson turns that into practice with numbers you can run: a fairness audit of a classifier, an adversarial attack and a saliency map on a NumPy network, and a short privacy demonstration. It ends with the model card, a document that records what a model is for and where it should not be used.

Five questions to ask

Fair? Does it work equally well for every group?
Private? Could it leak the people in its training data?
Robust? Does a tiny change flip its answer?
Explainable? Can we see what it looks at?

A fifth question: is it documented, with limits written down?

1 / 2

Bias starts in the data

Machine learning copies the patterns in its data, including the ones we would rather not repeat. Typical sources:

  • Sampling bias: a face dataset with few dark-skinned faces yields a model that is less accurate on them. Rare groups have little influence on an average loss.
  • Label bias: if the labels are past human decisions (who got hired, who was approved), the model learns to imitate those decisions, fair or not.
  • Measurement bias: the feature is a worse measurement for some people, for example a sensor, a test or a proxy variable that carries less signal for one group.
  • Feedback loops: a model decides who gets attention or credit, and those decisions become the next round of training data.

Removing a sensitive attribute such as gender from the inputs is not a fix. Other features (postal code, school, purchase history) are correlated with it, and the model can rebuild the information. The first step is to measure outcomes per group.

Measuring fairness

Split the evaluation data by group and compute the same metrics for each. Some common definitions, for a binary decision such as "approve a loan", where a positive prediction is the favorable outcome:

  • Selection rate: the fraction of each group that receives a positive prediction. Equal rates are called demographic parity.
  • True positive rate (TPR): of the people who deserve the positive outcome, the fraction who get it. Equal TPRs across groups are equal opportunity.
  • False positive rate (FPR): of the people who do not, the fraction who get it anyway. Equal TPR and FPR together are equalized odds.
  • Precision per group (also called predictive parity): when the model says yes, how often is that right?

These are computed from the confusion matrix covered in Evaluating Models, one group at a time. A famous result says that when the groups have different base rates of the true outcome, a non-perfect classifier cannot satisfy all of these definitions at once. So fairness is a choice about which error matters most in a given application, made with the people affected, and not a single number to maximize.

The example builds a synthetic population in which the input feature is a noisier measurement of true ability for group B (and slightly shifted). It trains a logistic regression on pooled data, audits it by group, and then applies a simple post-processing fix: a separate decision threshold per group that equalizes the true positive rate.

Python A fairness audit and a threshold fix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one shared threshold, the model is noticeably worse for group B, and the audit shows how: deserving members of B are approved only 51.8% of the time against 73.2% for A, and accuracy is 0.709 against 0.815, even though the training objective treated every example equally. Adjusting thresholds closes the TPR gap (0.741 against 0.724), but look at what it costs: the false positive rate of group B more than doubles from 0.156 to 0.335, its precision falls from 0.699 to 0.608, and its selection rate jumps from 0.305 to 0.502, now well above group A's 0.359. That trade-off is exactly the point above. No setting makes every gap vanish, so you must decide, document and monitor. The deeper fixes are usually upstream: collect better measurements and more data for the group the model serves worst, review the labels, and involve the people affected.

Fairness metrics are tools for finding problems, not a certificate. Passing one metric on one data split does not make a system fair, and legal and ethical requirements differ by country and domain.

Privacy

Models can remember. Large networks sometimes reproduce rare training examples, and membership inference attacks try to tell whether a given record was in the training set (usually because the model is more confident on data it has seen). Practical guidance:

  • Collect only what you need, remove direct identifiers, and keep a clear policy on what is retained.
  • Reduce overfitting; the gap between training and test behavior is what leaks (see Overfitting, Underfitting and Regularization).
  • Differential privacy gives a mathematical guarantee: the output of a computation changes very little whether or not any single person's data is included. It works by adding calibrated random noise. Training with DP-SGD clips each example's gradient and adds noise to the sum, trading some accuracy for a measurable privacy budget epsilon (smaller means more private).
  • Federated learning trains on devices and sends back updates instead of raw data; it reduces data movement but is not private by itself.

The simplest differential-privacy building block is the Laplace mechanism: to release a count, add noise of scale sensitivity / epsilon. One person can change a count by at most 1, so the sensitivity is 1.

Python The Laplace mechanism on a count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strong privacy (small epsilon) means noisy answers: the typical error on the count is about 0.07 at epsilon 10, 0.7 at epsilon 1 and 7 at epsilon 0.1. The noise is also relatively larger for small groups: at epsilon 1 a count of 5 is off by about 14% in the median, while a count of 5,000 is off by a negligible amount. That is a real design tension: the people in small groups are exactly those who are easiest to identify and whose statistics are least reliable.

Adversarial examples

A network can be fooled by changes too small for a human to notice. The fast gradient sign method (FGSM) shows why. A normal training step moves the weights downhill on the loss. An attacker moves the input uphill instead:

x_adv = x + epsilon * sign(dLoss/dx)

The gradient with respect to the input tells which direction of each pixel increases the loss most, and taking only its sign spends the same small budget epsilon on every pixel. Many tiny, coordinated changes add up to a large change in the output, particularly for the high-dimensional inputs of images. The same backward pass from Backpropagation in NumPy provides the input gradient, one step further than the first layer.

Attacks matter for spam filters, content moderation, fraud, and any system where someone benefits from fooling the model, and also for safety: a stop sign with stickers should not become a speed-limit sign. Defenses include adversarial training (training on attacked examples), input preprocessing, ensembles, and certified robustness methods, each with costs. Stay skeptical of claimed defenses, since many have later been broken by stronger attacks.

Explaining predictions with saliency

A saliency map shows which input features most affect a prediction. The simplest version is the absolute gradient of the predicted class score with respect to each input pixel: a large gradient means a small change at that pixel moves the score a lot. For a ReLU network, that gradient is computed by the same backward pass, with a one-hot output gradient.

Both ideas appear in the next example. It trains a network on the 8 by 8 digit images, measures the accuracy of clean and attacked inputs for growing epsilon, shows some attacked digits, and draws saliency maps.

Python FGSM attack and saliency maps on digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curve falls quickly with epsilon: accuracy goes from 0.919 on clean images to 0.524 when every pixel moves by at most 1.6 of 16 gray levels, and to 0.033 at a maximum change of 3.2 levels. The attacked digits still look like the originals to a person. The saliency maps are brightest on the strokes and nearby pixels that the network actually uses. Treat saliency with care: it is a local, first-order view, can be noisy, and different methods (integrated gradients, Grad-CAM, SHAP, LIME) can disagree. An explanation helps you debug, for instance by revealing that a model looks at a watermark instead of the object, but it is not proof that the model reasons the way you hope.

Model cards and documentation

A model card is a short document that travels with a model. Writing it forces the questions this lesson raised, and gives users and reviewers something to check. A useful card includes:

Section What to record
Intended use The task, the users, and the setting the model was built for
Out-of-scope use Where it should not be used (for example, medical decisions)
Training data Source, collection period, size, known gaps, preprocessing
Evaluation Metrics overall and per group, on which data
Limitations Known failure modes, robustness and uncertainty
Ethical considerations Risks, mitigations, privacy handling
Maintenance Owner, version, monitoring plan, how to report problems

Alongside the card, keep a datasheet for the dataset and the experiment records from Experiments and Reproducibility. Add human oversight for high-stakes decisions, an appeals path, and a monitoring plan like the one in Deploying and Serving Models. Finally, count the costs: training large models consumes real energy, which is one more reason to prefer the smallest model that does the job.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Removing the 'gender' column from the inputs guarantees that a model treats genders equally. Is that true?

    Proxies such as location or purchase history can carry the same information. Fairness is checked by measuring outcomes by group, not by hiding a column.
  2. 2. Equal opportunity asks for equal values of which quantity across groups?

    Equal opportunity means that people who deserve the positive outcome have the same chance of receiving it in every group, i.e. equal TPR.
  3. 3. What does FGSM do?

    It takes one step on the input in the direction that increases the loss the most, with every pixel changed by the same small amount.
  4. 4. A differentially private release of a count uses a small epsilon. What should you expect?

    The noise scale is sensitivity divided by epsilon, so a smaller epsilon means more noise and a stronger privacy guarantee.
  5. 5. Which statement about saliency maps is the most accurate?

    Saliency is a local sensitivity measure. It can reveal that a model looks at the wrong thing, but it is not a full explanation.

Summary

  • Bias comes from sampling, labels, measurement and feedback loops; dropping a sensitive column does not remove it.
  • Audit by group: selection rate, TPR, FPR and precision per group; different fairness definitions conflict when base rates differ, so choose deliberately and document it.
  • Post-processing such as per-group thresholds can reduce a gap but moves other errors; better data and measurements are the stronger fix.
  • Overfit models leak; differential privacy adds calibrated noise (scale sensitivity / epsilon) and trades accuracy for privacy.
  • FGSM perturbs inputs along the sign of the input gradient and quickly breaks an unprotected network; adversarial training helps but is not a guarantee.
  • Saliency maps show local sensitivity; use them to debug, not as proof.
  • Write a model card and a datasheet, keep humans in the loop for high-stakes use, and monitor after release.

Next: Project: Handwritten Digit Classifier.