Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 63 ยท Practice and Production

Evaluation, Bias and Responsible NLP

Learn how to build trustworthy NLP systems: split data safely, measure annotator agreement with kappa, audit bias by group, redact private data and analyze errors.

A model that scores 95% on your test set can still fail the people who use it. The test set may be too easy, too small or leaked from the training data. The labels may be inconsistent. The model may work well for one group of writers and badly for another. And the training text may contain private details that the model can repeat. Building a good NLP system means asking these questions on purpose, before users find the answers for you.

This lesson covers the habits that separate a demo from a system you can trust: sound datasets, labels people agree on, error analysis, bias audits, privacy and multilingual issues. Most of the techniques are small pieces of code that you can run on any project, and we practice them on toy data you can change.

Data first

Questions about your data

Representative? Does it look like real user text?
Labeled consistently? Would two people give the same label?
Leaked? Do test examples appear in training?
Balanced? Are all groups and classes present?

Most NLP quality problems start here, not in the model.

1 / 2

Some practical rules:

  • Split before you look. Keep a test set that you do not use for tuning (the habit from classification). Tune on a separate validation set or with cross-validation.
  • Beware of near-duplicates. Scraped text is full of copies, retweets and boilerplate. If a sentence appears in both training and test sets, your score is inflated. Deduplicate before splitting, and split by document or author (not by sentence) when sentences from one source are alike.
  • Split by time for time-based tasks. If the model will predict tomorrow's tickets, test on newer data than you trained on.
  • Match production. If users write on phones with typos and emoji, a test set of clean news articles tells you little.
  • Choose metrics that fit the cost of mistakes. Accuracy misleads on imbalanced data; use precision, recall and F1 per class, and look at the confusion matrix.
  • Document the dataset: where the text came from, how it was labeled, which languages and groups it covers, and what it must not be used for.

Do annotators agree? Cohen's kappa

Labels come from people, and people disagree. If two annotators label the same items and agree only 70% of the time, no model can reliably score above that on those labels, because the "truth" itself is unclear. Inter-annotator agreement tells you how clear the task is.

Raw agreement is misleading when one class dominates: if 90% of reviews are positive, two annotators who both mostly say "positive" will agree often by luck. Cohen's kappa corrects for this by subtracting the agreement expected by chance:

TEXT
kappa = (observed - expected) / (1 - expected)

A kappa of 1 means perfect agreement, 0 means no better than chance. As a rough guide, values above 0.8 are considered strong, 0.6 to 0.8 substantial, and below 0.4 weak, though the right threshold depends on the task. The code computes kappa for two annotators on skewed and balanced data, and for a model that always predicts the majority class.

Python Cohen's kappa by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both annotator pairs agree on 90% of the items, yet their kappas differ widely. On the skewed data, 82% agreement is expected by chance, so kappa is only 0.44: the annotators barely beat luck on the hard minority class. On balanced data, chance agreement is 50% and the same 90% gives a kappa of 0.80. The always-positive model reaches an accuracy of 0.90 and a kappa of 0.00: it has learned nothing. When kappa is low, fix the annotation guidelines (clear definitions, examples of borderline cases) and have disagreements discussed, before you spend time on modeling. For more than two annotators, use Fleiss' kappa or Krippendorff's alpha.

Error analysis

A single score hides what goes wrong. After each experiment, read your mistakes: print 50 wrong predictions and sort them into categories (negation, sarcasm, rare words, typos, a label that looks wrong, an ambiguous example). Counting the categories tells you what to fix next, whether it is more training data of one kind, a preprocessing step, or a cleaner label. Often a good share of "errors" turn out to be label mistakes, which is a data fix rather than a model fix. Also compare confusion pairs (which true class is mistaken for which) and look at the examples the model was most confident about and got wrong.

Bias: does it work for everyone?

Models learn the patterns in their training text, including social biases. Word embeddings place some occupations nearer to one gender than another; sentiment and toxicity classifiers can rate sentences mentioning certain identities, dialects or names as more negative; speech and translation systems often work better for well-represented languages and accents. These are not abstract problems: a biased moderation model can silence some communities, and a biased rรฉsumรฉ screener can reject qualified people.

You cannot fix what you do not measure, so evaluate by slice: compute the same metrics separately for each group or kind of text that matters, and compare them. Large gaps in accuracy, or in error types such as the false-positive rate, are a red flag. The next example uses an invented toxicity test set (not real data) to show the mechanics. Texts are grouped by writing style, and the aggregate score hides a gap.

Python Slice-based bias audit on invented data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overall accuracy is 0.75, which sounds acceptable, but by group it is 1.00 for formal text and 0.50 for informal text. The false-positive rate (harmless comments flagged as toxic) is 0.00 against 0.60. All three mistakes are of one type, a harmless sentence flagged as toxic, and all of them fall on informal writing. Reading the wrong examples reveals the cause: ordinary informal grammar looks unfamiliar to a model trained mostly on formal text. This pattern has been documented in real systems, and the usual remedies are better and more varied training data, evaluating by group before launch, adjusting thresholds per use case, and keeping a human review path for contested decisions. With six examples per group these numbers are only an illustration; real audits need hundreds of examples per slice and confidence intervals.

Privacy and sensitive data

Text often contains personal data: names, emails, phone numbers, addresses, health details. Rules differ by country (GDPR in Europe, for example), but the engineering habits are universal: collect only what you need, redact before storing or sharing, restrict access and delete on request. Language models can memorize rare strings from their training data and repeat them, so never train on data you are not allowed to expose. A first line of defense is pattern-based redaction, covered by rule-based extraction. The code redacts emails, card numbers and phone numbers, and also shows its limits.

Python Redacting personal data with regex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The email, phone number and card number are replaced by placeholders. The order of the patterns matters: if the phone pattern ran before the card pattern, it would swallow the card number and label it wrongly. The third sentence is the lesson: the name Jordan Smith and the place Berlin pass through untouched, because names have no fixed pattern. Catching them takes a named entity recognizer, which is itself imperfect, so serious pipelines combine rules, models and human review, and measure what slips through.

Multilingual and cross-domain issues

English dominates NLP resources. Many languages have far less data, fewer tools and weaker models, and results reported for English rarely transfer. Keep these in mind:

  • Evaluate in each language you serve, and per script and dialect; a multilingual model's average score hides weak languages.
  • Tokenization costs differ. Languages that are split into many subword tokens cost more per API call and fit less text in a context window.
  • Languages without spaces (Chinese, Japanese, Thai) need different tokenization, and rich morphology (Finnish, Turkish) hurts word-level counts.
  • Code-mixing (for example, Hindi and English in one message), transliteration and slang appear in real data but rarely in benchmarks.
  • Domain shift: a model trained on movie reviews can fail on medical notes or legal text. Test on the domain you deploy to.
  • Time drift: language changes, new products and words appear, and accuracy decays; the next lessons on production cover monitoring for that.

A responsible-NLP checklist

  1. Define who will use the system, what harm a mistake could cause and what the fallback is.
  2. Document data sources, labeling instructions, annotator agreement and known gaps.
  3. Split without leakage; keep a final test set untouched.
  4. Report per-class and per-slice metrics, not only the average.
  5. Read errors, count categories and fix data before models.
  6. Remove or protect personal data; check that the model cannot leak it.
  7. Be transparent: tell users when they are talking to a model, and give them a way to contest decisions.
  8. Monitor after launch, and keep a human in the loop for high-stakes use.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is Cohen's kappa preferred over raw agreement for comparing annotators?

    When one class dominates, annotators agree often by luck. Kappa subtracts the expected chance agreement, so it shows how much better than chance they really are.
  2. 2. Your data contains many near-duplicate sentences from the same sources. What is the best way to split it into train and test sets?

    If near-copies appear on both sides, the model is tested on text it has effectively seen, so the score is inflated. Grouping by source avoids this leakage.
  3. 3. A classifier has 90% accuracy overall, but a 50% accuracy on text from one group. What should you do?

    Averages hide groups that the system fails. Slice metrics expose the gap, and reading the errors shows what to fix, usually data coverage.
  4. 4. Why did our regex redaction fail to remove "Jordan Smith"?

    Emails and card numbers have a regular shape, but names do not. Detecting them takes statistical models, and even those miss some.
  5. 5. Why should you evaluate a multilingual system in each language separately?

    Resources are very uneven across languages, so an average can look fine while some languages are served badly. Per-language metrics make that visible.

Summary

  • Trust starts with data: representative, consistently labeled, deduplicated, split without leakage and documented.
  • Cohen's kappa measures annotator agreement beyond chance; low kappa means the task or guidelines need work.
  • Error analysis (reading and categorizing mistakes) tells you what to fix; many errors are label or data problems.
  • Audit bias by slicing metrics per group; an average can hide a large gap.
  • Protect privacy: collect less, redact with rules and models, restrict access, and never assume text is safe to train on.
  • Check each language and domain you serve, and keep monitoring after launch.

Next, take a model into the real world in NLP in Production.