Skip to content
Tutorials Center
My learning

Topics

Lesson 84 of 111 ยท Beyond the Basics

Fairness, Bias and Responsible ML

See how sampling bias, label bias and proxy features create unfair models, measure group metrics by hand, weigh the trade-offs and document and protect your data.

A model learns from data, and data records what people did, including their mistakes and unfair habits. A hiring model trained on past hiring decisions can learn to repeat them. A medical model trained mostly on one population can quietly work worse for everyone else. None of this requires bad intent: it happens by default unless someone looks for it.

This lesson is a practical introduction to looking. You will build a small synthetic loan-approval dataset where we control the truth, watch bias enter through labels, sampling and proxy features, measure the damage with group metrics written by hand, and see why "fixing" it always involves trade-offs. Real fairness work also needs law, domain experts and the people affected. Code alone cannot settle what is fair.

Where unfairness enters

Sampling bias Some groups barely appear in the data.
Label bias Past decisions were unfair, and the labels copy them.
Proxies Other columns stand in for a protected one.
Deployment A model used in a new setting behaves differently.
1 / 2

Overall accuracy hides groups

Suppose a model is 90% accurate overall. If 95% of your rows come from one group, the model could be perfect for them and nearly useless for the other 5%, and the overall number would hardly move. The first rule of responsible evaluation is therefore to slice every metric by group: the same precision, recall and error rates you know from Classification Metrics, computed separately for each group that matters (age bands, regions, genders, devices, languages).

A few group-level quantities come up constantly:

Quantity Meaning
Selection rate Share of the group the model says "yes" to (approved, hired, flagged)
True positive rate (TPR) Of the people who truly deserved "yes", the share who got it
False positive rate (FPR) Of the people who truly deserved "no", the share who got "yes"
Precision Of the people who got "yes", the share who truly deserved it

Two simple fairness criteria use them. Demographic parity asks for equal selection rates across groups. Equal opportunity asks for equal TPR: qualified people should have the same chance of approval whichever group they belong to. Neither is "the" correct definition, as you will see.

A dataset where we know the truth

We build loan applicants in two groups, A (70%) and B (30%). Each person has a hidden skill, which decides whether they are really qualified to repay. By construction, qualification does not depend on group. Past decisions, however, were biased: group B applicants were approved only with a much stronger record. We observe a test_score (a noisy copy of skill) and a neighborhood score that is correlated with the group.

In real life you never know the true qualification, only the past decision. Here we know both, and that lets us see what the bias does.

Python Label bias: past decisions versus true qualification

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two groups have almost the same true qualification rate (about half), yet group B was approved far less often. A model that treats approved_past as the truth would inherit this gap. That is label bias: the labels measure a human decision, not the reality we care about.

Group metrics by hand

To audit a model, write a small function that computes the quantities above from predictions, truth and group. It uses only NumPy and pandas, so you can see exactly what each number means. We start with a model that sees only test_score and is trained on the biased labels.

Python A group audit function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only the test score, the two groups are treated about alike. Group B's historical penalty applied mainly at the very top of the score range, so a simple model that sees only the score barely notices it. That looks reassuring, but the next section shows how fragile it is.

Proxies: removing the column is not enough

A tempting fix is to delete the sensitive column (group) and call the model fair. This is fairness through unawareness, and it fails whenever other columns carry the same information. Postal code, first name, school, shopping habits or device type can all act as proxies. A model never needs to see the group to learn its effects: it only needs a column that is correlated with the group and with the biased label.

Python Adding a proxy brings the bias back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the two tables side by side. With the proxy and the biased labels, group B gets a lower selection rate and a lower TPR: qualified people in B are turned down more often than qualified people in A, even though group never entered the model. Trained on the truth, the same features give both groups similar numbers, because the proxy carries no real signal about qualification.

The real lesson is the cause, not the fix: the unfairness came from the labels. In practice you rarely have a clean "true qualification" column, so you must question what the target really measures. Is "was hired" a measure of ability, or of who recruiters liked? Is "was arrested" a measure of crime, or of where police patrolled?

Keeping a sensitive column out of the model hides it from the model, but also from you. You cannot audit what you did not record. Collect group information (carefully, lawfully, with consent where required) in a separate table and use it for evaluation, not as an input feature, unless the law and the use case say otherwise.

Sampling bias

Another route to unfairness is who appears in the data. If a group is rare in training, the model optimizes mostly for the others. If the group's relationships also differ (for example, a symptom signals a condition differently in another population), the model will be systematically worse for them.

The simulation below keeps the test set fixed (50/50 groups) and varies how many group B rows the training set contains.

Python Accuracy for a group follows its share of the training data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only a sliver of group B in training, the model learns group A's boundary and group B pays for it. More B data helps, but a single linear model has to compromise between two different boundaries, which is why collecting representative data comes first, and why sometimes a separate model, or a feature that encodes the difference, is justified.

Trade-offs: you cannot satisfy every definition

Suppose the audit found a gap. Many remedies exist: change the labels, collect better data, reweight rows, drop the proxy, or adjust thresholds per group after training. The last one is easy to try. The next example chooses a separate threshold for each group so that both reach the same TPR (equal opportunity), and shows what it costs.

Python Per-group thresholds to close the TPR gap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The adjustment gives group B a lower threshold and shrinks the TPR gap from about 0.10 to about 0.02. Other numbers move too: group B's precision falls a little (0.83 to 0.81) because more of its approvals are mistakes. Here overall accuracy happens to rise slightly, because the original model was imitating biased labels; do not count on that, since the usual shape of the fairness trade-off is that something gets worse.

Three things worth knowing:

  • The criteria conflict. When two groups have different base rates of the true outcome, a model that is imperfect cannot have equal selection rates, equal TPR and equal precision all at once. You must choose what matters in your setting (a lender, a hospital and a search engine will choose differently) and say so openly.
  • Group-specific thresholds may be illegal or unacceptable in some domains, because they treat people differently by group. Check the law before using them; this example only shows the effect.
  • Fixing the labels or the data usually beats post-processing. Adjusting outputs patches a symptom; cleaning the target addresses the cause.

Documentation and accountability

Responsible ML is also paperwork, and that is a feature, not a chore. Two lightweight documents are widely used:

  • A datasheet for the dataset: who collected it, when and why, which populations are covered or missing, how labels were created and by whom, and known problems.
  • A model card for the model: its intended use, the uses it was not built for, the evaluation data, metrics broken down by group, known limitations and who to contact.

Write the model card while you evaluate, not after launch. If you cannot fill in the "metrics by group" section, you have found a gap in your evaluation. Beyond documents, plan for human review of high-stakes decisions, a way for people to appeal an outcome, and ongoing monitoring (see Saving Models and Deploying).

Privacy basics

Models learn from personal data, so protecting that data is part of responsible practice. The core habits:

  • Minimize. Collect only the columns you need. A column that is never stored cannot leak.
  • Remove direct identifiers (names, emails, ID numbers) before analysis, but remember that removing them does not make data anonymous.
  • Beware quasi-identifiers. Birth date, postal code and gender together can single out a person in many datasets. A simple check is k-anonymity: every combination of such columns should be shared by at least k people.
  • Models can leak. Overfit models can memorize rare training rows, and publishing a model can reveal more than you intend. Techniques such as differential privacy exist, but start with minimization and access control.
  • Follow the rules. Laws such as GDPR govern consent, purpose and deletion rights. Involve whoever is responsible for them early.
Python A k-anonymity check by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Anyone whose combination appears only once (group_size 1) could be re-identified by someone who knows those three facts about them, and then their diagnosis is exposed. Generalizing columns trades detail for protection, a trade-off that mirrors the fairness trade-offs above.

A short checklist

  1. Ask what the label really measures and who produced it.
  2. Check who is in the data and who is missing.
  3. Slice every metric by group, including small groups, and look at the confidence of the estimates (few rows means noisy numbers).
  4. Look for proxies: can the sensitive attribute be predicted from the features?
  5. Pick the fairness criterion deliberately, write down why, and accept the cost.
  6. Document, review and monitor after launch.
  7. Decide whether to use ML at all for this decision (see What Is Machine Learning?).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A model has 94% overall accuracy, and one group makes up 3% of the data. What is the best next step?

    Overall accuracy is dominated by large groups and can hide poor performance on small ones. Slice the metrics by group.
  2. 2. Why does deleting the 'group' column not guarantee a fair model?

    A feature such as neighborhood can be correlated with the group. If the labels are biased, the model can learn the bias through that proxy.
  3. 3. What does the equal opportunity criterion require?

    Equal opportunity asks that people who truly deserve a positive outcome have the same chance of getting it in each group, which means equal TPR.
  4. 4. Which situation is an example of label bias?

    When the label measures a biased decision instead of the true outcome, any model trained to imitate it learns the same bias.
  5. 5. In a k-anonymity check, what does 'the data set is 1-anonymous' tell you?

    k is the size of the smallest group sharing the same quasi-identifier values. With k equal to 1, someone is unique and could be re-identified.

Summary

  • Models inherit bias from sampling, from labels and from proxy features; none of it needs bad intent.
  • Slice every metric by group: selection rate, TPR, FPR and precision, which you can compute by hand from the confusion counts.
  • Removing a sensitive column does not remove its effects when proxies exist; test whether the attribute can be predicted from the remaining features.
  • Fairness criteria such as demographic parity and equal opportunity can conflict. Choose one on purpose, explain why and measure its cost.
  • Fixing labels and collecting representative data beats patching outputs with per-group thresholds.
  • Document datasets and models, plan for review and appeals, minimize data and check quasi-identifiers (k-anonymity).

Next up: learn the mistakes that sink real projects in Common Machine Learning Mistakes.