Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท Governance and Cost

Data Catalogs, Metadata and Lineage

Learn what a data catalog stores, the types of metadata, ownership and data products, and how lineage graphs answer impact and root-cause questions.

When a company has 5,000 tables, the hardest question is no longer "how do I query this?" but "which table should I trust, who owns it, and what breaks if I change it?". Data governance is the set of policies and practices that answer those questions. Its day-to-day tools are the data catalog (a searchable inventory of data assets) and lineage (a map of how data flows).

Commercial and open-source catalogs (DataHub, OpenMetadata, Unity Catalog, Collibra, Atlan, AWS Glue Data Catalog) are services you deploy. The code in this lesson is a small, labelled stand-in that models what they store, so you can see the ideas without installing anything.

What a catalog stores: metadata

Metadata is data about data. Catalogs usually hold four kinds:

Kind Examples Source
Technical schema, types, partitions, size, row count, location, last update harvested automatically
Business description, glossary terms ("active customer"), domain written by owners
Operational freshness, run history, quality results, usage pipelines and query logs
Social / governance owner, steward, tags (PII, finance), certification people and policies

A good catalog answers in seconds: what is this column, who owns the table, is it certified, how fresh is it, who uses it, and what feeds it.

A catalog entry as data

Python Stand-in: a tiny catalog with search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real catalogs add column-level descriptions, popularity ranking from query logs, and a UI, but the shape is the same: structured records plus search.

Lineage: how data flows

Lineage records which inputs produced each output. It exists at several levels: table to table, column to column, and job to dataset. Together, the edges form a graph:

flowchart LR
  A["raw.orders"] --> B["stg.orders"]
  C["raw.customers"] --> D["stg.customers"]
  B --> E["core.orders"]
  D --> E
  E --> F["mart.daily_revenue"]
  F --> G["Finance dashboard"]

Lineage is captured by parsing SQL (dbt manifests, query logs), by instrumenting engines (OpenLineage events from Spark and Airflow), or by hand for systems that cannot be parsed.

Two questions lineage answers

  • Impact analysis (downstream): "If I rename core.orders.amount, what breaks?"
  • Root cause analysis (upstream): "The dashboard number is wrong; which sources feed it?"
Python Walk the lineage graph in both directions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Column-level lineage goes further: "mart.daily_revenue.revenue = core.orders.amount minus core.refunds.amount". It is what lets you find every report that uses a PII column or a deprecated field.

Ownership and data products

A catalog without owners decays. Practical rules:

  • Every table in a curated layer has a named owner (a team, not just a person) and a steward for definitions.
  • A data product is a dataset treated like a product: clear owner, documented schema and semantics, quality guarantees (freshness, completeness), support channel and versioning. The "data mesh" approach organizes ownership around domains that publish such products.
  • Use tiers or certification: bronze/silver/gold, or "certified" versus "experimental", so analysts know what is safe for executive reporting.
  • Add deprecation metadata: replaced-by, sunset date, and warn consumers through lineage.

A business glossary

Different teams define "active user" differently, and dashboards disagree. A glossary gives canonical definitions and links them to columns and metrics. When "revenue" has one definition owned by finance, arguments become fixes to one definition rather than to twelve dashboards. A semantic layer (metrics defined once in code, such as dbt metrics or Cube) enforces the definition at query time.

Python Find undocumented or unowned assets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Governance scorecards like this (percent of tables with an owner, description and tests) are a simple way to measure progress.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which is technical metadata?

    Technical metadata describes structure and physical properties and is harvested automatically.
  2. 2. Which question is best answered by downstream lineage?

    Downstream lineage (impact analysis) lists everything that depends on an asset.
  3. 3. What is a data product?

    Treating data as a product adds ownership, documentation, SLAs and a support path.
  4. 4. Why is a business glossary useful?

    A shared definition prevents conflicting numbers across teams.

Summary

  • A catalog is a searchable inventory of technical, business, operational and governance metadata.
  • Lineage graphs support impact analysis and root-cause analysis; capture them from SQL parsing or OpenLineage events.
  • Name owners, certify trustworthy assets, define terms once and measure documentation coverage.

Next: Privacy, PII and Access Control