Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Integration and Data

Data Quality Monitoring and Governance

Define data quality dimensions, write validation rules, compute quality scores and set up ongoing monitoring and stewardship for ERP and CRM data, with Python rule engines you can run.

A migration can deliver clean data on day one, and a year later the data is dirty again: new duplicates, missing phone numbers, outdated addresses, products with no weight. Data decays because people are busy, forms allow free text and systems feed each other mistakes. Data quality management is the ongoing practice of defining what "good" means, measuring it continuously and fixing problems at the source.

This lesson covers quality dimensions, rule-based checks, scoring and the governance routines that keep scores high. The ERP and CRM products are not available here, so the checks run as Python stand-ins.

Dimensions of data quality

Is it there and right?

Completeness Required values are filled in.
Accuracy Values match the real world.
Validity Values follow formats and rules.

The first things most checks cover.

1 / 2
  • Completeness: a customer has an email; a product has a weight and a unit.
  • Validity: the email has a valid shape; the country is a real ISO code; dates are real dates.
  • Accuracy: the address actually exists. Hardest to measure, because it needs an external truth (a postal database, a call).
  • Consistency: the order currency matches the customer's currency; the same customer has the same tax ID in CRM and ERP.
  • Uniqueness: no duplicates, covered in master data management.
  • Timeliness: the last contact was this year; exchange rates are today's.

Turning rules into checks

A data quality rule is a testable statement such as "every active customer has a country code from the ISO list". Each rule has an ID, a dimension, a severity, a business owner and an automatic check. Rules should come from real business harm: "missing tax ID blocks invoices to EU customers".

Python A small rule engine (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each failure is traceable to a record and a rule, which tells the steward exactly what to fix.

A single quality score per domain makes the topic visible to management. A simple measure: the percentage of records that pass each rule, and an overall weighted average. Track it monthly. A score by itself matters less than its direction and its distribution across teams or regions.

Python Quality scorecard with weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fix at the source

Cleaning data downstream (in the warehouse or a spreadsheet) only hides the problem; the next load brings it back. Better:

  • Prevent: mandatory fields, drop-down lists instead of free text, format checks, address validation at entry, duplicate checks before saving.
  • Detect: scheduled rule runs, dashboards, alerts when a score drops under a threshold.
  • Correct: work queues for stewards, with the failing records, the rule and a suggested fix.
  • Learn: analyse the root cause. If one sales team creates most bad records, the fix is training or a better form, not more cleaning.

Be careful not to add so many mandatory fields that users type junk ("n/a", "xxx") just to proceed. Junk values pass completeness checks while failing accuracy.

Python Spot placeholder values that fake completeness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Governance routines

Quality does not maintain itself. Set up:

  • A data governance council that sets policies and settles conflicts between departments.
  • Data owners and stewards for each domain with defined response times.
  • A data catalog or dictionary with definitions: what exactly is an "active customer"?
  • Regular reviews: monthly quality scorecards, quarterly rule reviews, annual retention checks.
  • Access and privacy controls: who may see and change data, retention periods, deletion on request where privacy law applies. Quality and compliance are linked: personal data that is outdated or kept too long is both a quality and a legal problem.
  • Change control for master data structures: new fields and values go through a request process.

Feed quality findings into the analytics programme too, because reports are only as good as the data below them (see reporting, KPIs and dashboards).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A customer record has an email value 'n/a'. Which dimension is most clearly broken, even though the field is not empty?

    The field looks filled but the value is not a valid or real email, which a simple not-empty check would miss.
  2. 2. Why is fixing data only in the reporting layer a weak strategy?

    Root causes in entry forms, integrations or processes keep creating bad data unless they are fixed at the source.
  3. 3. What is a data steward's main job?

    Stewards work through failing records, duplicates and requests, following the owner's policies.
  4. 4. Which is the best example of a data quality rule?

    A good rule is specific, measurable per record and linked to a business consequence.

Summary

  • Quality has dimensions: completeness, validity, accuracy, consistency, uniqueness and timeliness.
  • Write rules tied to business harm and run them automatically.
  • Score and trend quality, and watch for junk values that fake completeness.
  • Prevent at entry, detect with dashboards, correct through steward queues and fix root causes.
  • Governance gives data owners, stewards, definitions and review routines.

Next: KPIs, Dashboards and Business Reports