Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 76 ยท Ethics, Privacy and Judgement

Ethics, Fairness and Responsible Data Use

Learn privacy and consent basics, why anonymised data can be re-identified, how proxy variables leak group bias, and how to compare error rates across groups.

Data science turns data about people into decisions about people: who gets a loan, which job applications are read, which patients are called back. Mistakes then cost real people something, and they are rarely spread evenly. A model that is right 90% of the time can be right 97% of the time for one group and 70% for another. Averages hide that, and a project that never looks has not shown the problem is absent.

This lesson is not a list of rules to memorize. It gives you habits and a few tools you can run: a test for how identifiable a table is, a way to see whether a "neutral" feature secretly stands in for a protected group, and a group-by-group error comparison. Law differs between countries (the GDPR in Europe, various state laws in the US, and others), and this lesson is not legal advice. If a project affects people's rights or money, involve your legal and privacy teams early.

flowchart LR
    A["Purpose and consent"] --> B["Data minimisation<br/>and privacy"]
    B --> C["Model and features"]
    C --> D["Group-wise evaluation"]
    D --> E["Harm review<br/>and human oversight"]
    E --> F["Monitor after launch"]

Responsible practice is a thread through the whole project, not a final checkbox.

Start with three questions before you collect or reuse any data:

  • Purpose: what exactly will this data be used for? People who agreed to share their address for deliveries did not agree to have it used for pricing.
  • Consent and expectations: would the people in the data be surprised by this use? If yes, you need a stronger justification, or a different design.
  • Minimisation: do you need this column at all? Data you do not collect cannot leak, be misused or be subpoenaed. Drop fields that do not serve the stated purpose, and keep personal data only as long as needed.

Also ask who might be missing from the data. If a health app's users are younger and richer than the population, conclusions drawn from it may not hold for everyone (see Sampling and Collection Bias).

Anonymisation is harder than removing names

Deleting the name column does not make a dataset anonymous. Combinations of ordinary-looking columns can single out a person. Postal code, birth year and gender are called quasi-identifiers: none identifies you alone, but together they often do. A famous result is that a large share of a country's population is unique on just these three. Anyone who also holds a voter list, a social media profile or a leaked table can link the records.

A simple measure is k-anonymity: a table is k-anonymous if every combination of quasi-identifiers appears at least k times. If a combination appears once, that row is unique and easy to re-identify. You can count it directly.

Python How identifiable is a table? Counting unique combinations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Generalizing (a coarser zip, decade bands) cut the share of unique rows from 74% to under 1%, at the cost of detail. That trade-off is the heart of privacy work: more utility means more risk. The few rows that stay unique sit in the rare gender category, which is why teams also suppress small cells in published tables, or generalize that column too.

Remember what k-anonymity does not cover. Even when you cannot tell which row is a person, everyone in a group may share the same diagnosis, which reveals it anyway. Stronger methods such as differential privacy add calibrated noise so that no single person's presence changes results much. They are the tools for publishing statistics from sensitive data, and worth learning if your work needs them.

Warning: Treat "we removed the names" as a claim to test, not as a fact. Hashing an e-mail address or ID does not anonymise it either: anyone can hash a list of known addresses and match them.

Proxy variables: bias without the protected column

Many teams respond to fairness concerns by deleting the protected attribute (gender, ethnicity, religion, age, disability) from the training data. This rarely works. Other columns often carry the same information. A postal code can stand in for ethnicity or income; a first purchase category can hint at gender; the time of day an application was sent can hint at shift work. Such columns are proxy variables.

You can look for proxies by asking how well the other features predict the protected attribute. If a model can recover it, so can your main model.

Python A model without the group column still uses it through a proxy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two groups have the same skill distribution, and the model never saw the group. Yet the selection rates differ, because the neighborhood score carries group information and the historical labels it learned from were skewed. Removing a column is not the same as removing its influence. This is the sense in which a model can inherit past discrimination: it is faithful to biased history.

Fairness metrics: comparing groups

There is no single definition of "fair"; there are several measurable properties, each answering a different question. Compute them per group from the confusion matrix:

Measure Question it answers
Selection rate (share predicted positive) Do groups get the favourable outcome at similar rates? (demographic parity)
True positive rate (recall) Among people who deserve the outcome, are groups found equally often? (equal opportunity)
False positive rate Among people who do not, are groups wrongly flagged equally often? (with TPR: equalized odds)
Precision (positive predictive value) When the model says yes, is it right equally often in each group?
Calibration by group Does a score of 0.7 mean the same thing in each group?

A hard fact: when groups have different base rates of the outcome, these properties cannot all be satisfied at once (except in trivial cases). You have to choose which error matters most for your context and say so. A false positive in fraud screening (an innocent person blocked) harms differently than a false positive in medical triage (an extra test). The choice is a decision about values, and should be made by people accountable for it, not hidden in a default threshold.

Python Group-wise error comparison and the effect of thresholds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the two tables. With one threshold, the groups get different error rates because the main measurement is noisier for group B: its false positive rate is higher (0.25 against 0.19) and its precision lower (0.62 against 0.67), while its true positive rate is slightly lower (0.47 against 0.49). Raising group B's scores by 0.1 lifts its true positive rate to 0.65, now above group A's, but its false positive rate jumps to 0.40 and the gap in false positives grows. That is the trade-off made visible: moving one measure moves the others. There is no free fix, which is exactly why you must look, decide and document.

Two cautions. Using the group to set thresholds is itself restricted or illegal in some domains and countries (for example in credit and employment decisions), so check what is allowed before using it as a remedy. And be wary of small groups: a TPR of 0.62 measured on 40 people has a wide uncertainty, so report confidence intervals (see Estimation and Confidence Intervals) rather than reading too much into gaps.

Harm analysis in practice

Metrics help, but they come after a more basic exercise: thinking about who can be hurt and how.

  1. List the stakeholders: the people scored, the people who act on the scores, the organization, and bystanders.
  2. List the harms for each wrong outcome: denial of a service, extra scrutiny, lost income, stigma, wasted time. Include harms that are hard to appeal or reverse.
  3. Ask who bears the cost of errors. Is it the same group that benefits when the model is right?
  4. Plan oversight and redress: can a person review a decision? Can the affected person see why and contest it? Is there a human fallback when the model is uncertain?
  5. Choose what to measure, publish and monitor: group-wise metrics at launch and every month, with named owners.

Document the result in a short "model card": purpose, data, known limits, performance by group, and intended and unintended uses. It is also the best defence against a model being quietly reused for a purpose it was never validated for.

Other habits worth having

  • Do not treat a model score as truth. Scores are estimates; present them with uncertainty and keep humans in the loop for high-stakes cases.
  • Beware feedback loops. If a model decides where to inspect, it finds problems there and confirms itself, while problems elsewhere stay unseen.
  • Be honest in communication. Say what the data cannot show; do not stretch correlations into causes (Causal Thinking); avoid charts that exaggerate (see Principles of Effective Visualization).
  • Share unwelcome findings. If your analysis shows a harm, reporting it is part of the job.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is deleting the name column not enough to anonymise a dataset?

    Each quasi-identifier is harmless alone, but together they are often unique, and they can be linked with other data sources.
  2. 2. A table is 5-anonymous. What does that mean?

    k-anonymity says no combination of quasi-identifiers occurs fewer than k times. It does not protect against everything, for example when all members of a group share the same sensitive value.
  3. 3. You remove the protected attribute from training data, but outcomes still differ by group. What is the most likely reason?

    Features such as location can predict the group. A model trained on biased historical decisions will reproduce them through those proxies.
  4. 4. Why can you usually not make selection rate, true positive rate and precision equal across groups at once?

    With different base rates and an imperfect model, equalizing one measure shifts the others. You must decide which matters most for the context.
  5. 5. What should accompany a group-wise TPR of 0.62 measured on only 40 people?

    Small groups give noisy estimates. Report uncertainty so that gaps are not over-interpreted, and do not hide the group.

Summary

  • Define purpose, consider consent and expectations, and collect only what you need.
  • Removing names is not anonymisation; measure uniqueness (k-anonymity), generalize, and suppress small groups, knowing the limits of those methods.
  • Proxy variables let a model use a protected attribute without being given it; test whether features can recover it.
  • Compare error rates by group (selection rate, TPR, FPR, precision, calibration); different fairness definitions conflict when base rates differ, so choose with the context in mind and write the choice down.
  • Do a harm analysis, plan human oversight and redress, document the model, and keep monitoring after launch.
  • Check what the law and your organisation allow before applying group-specific fixes.

Next: put the whole toolkit to work in Project: Customer Churn Analysis.