Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Governance and Cost

Privacy, PII and Access Control

Protect sensitive data in pipelines: classify PII, mask and pseudonymize with hashing, apply least privilege, row and column security, and handle deletion requests.

A data platform concentrates the most sensitive information a company owns: names, emails, payment data, health records, location trails. Mistakes here lead to breaches, fines (GDPR, CCPA/CPRA, HIPAA) and lost trust. Data engineers are on the front line, because pipelines copy data into many places, and every copy needs protecting.

This is an engineering introduction, not legal advice. Rules vary by country and industry; involve your privacy and security teams.

Classify first

You cannot protect what you have not identified. Tag columns by sensitivity:

Class Examples Handling
Public product names no restriction
Internal revenue by region employees only
Confidential / PII email, phone, address, IP restricted, masked by default
Highly sensitive national id, card number, health data strongest controls, minimal copies, often tokenized

Direct identifiers (name, email) identify a person alone. Quasi-identifiers (birth date, ZIP code, gender) identify someone when combined: a famous study showed that most people in a population are unique by birth date, gender and ZIP. So "removing the name" is not anonymization.

Python Scan columns for PII patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pattern scanners find obvious cases. Combine them with owner review, because free-text fields and screenshots hide PII in surprising places.

Masking, hashing and tokenization

Technique What it does Reversible? Use
Redaction / masking [email protected] becomes a**@example.com no showing data to analysts
Hashing with a secret (HMAC) stable pseudonym per value no (without brute force on small domains) joining across datasets without the raw value
Tokenization replace with a random token, mapping in a vault yes, via the vault payment cards, ids needed later
Encryption ciphertext with a key yes, with the key storage at rest, per-column
Generalization exact age 37 becomes bucket 30-39 no reducing re-identification risk
Python Mask, pseudonymize and generalize

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why HMAC and not a plain SHA-256 of the email? Emails come from a small, guessable space, so attackers can hash a list of known addresses and match them. A secret key makes that infeasible. Remember that pseudonymized data is still personal data under GDPR, because it can be linked back by whoever holds the key or mapping.

Least privilege and RBAC

Grant the minimum access needed. Use role-based access control (RBAC): people get roles (analyst, finance, engineer), roles get privileges on datasets. Avoid granting to individuals.

Common warehouse controls:

  • Column-level security / dynamic masking: analysts see masked email unless their role is "pii_reader".
  • Row-level security: a regional manager sees only rows of their region.
  • Views as an access layer: expose a view that hides sensitive columns instead of the base table.
  • Separate environments and accounts for production and development; use synthetic or masked data in dev.
Python Stand-in: role-based column masking and row filter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real warehouses enforce this declaratively (masking policies in Snowflake, policy tags in BigQuery, Unity Catalog column masks), so rules live in one place and apply to every query tool.

Protecting data in motion and at rest

  • Encrypt in transit (TLS) and at rest (cloud-managed keys, or customer-managed keys for stricter needs).
  • Secrets: keep credentials in a secrets manager; never commit them to repositories or DAG files; rotate regularly.
  • Network: private endpoints, no public buckets, restricted egress.
  • Audit logs: record who read which sensitive dataset and when; review and alert on unusual access.

Retention and deletion requests

Privacy laws grant rights such as access and erasure ("right to be forgotten"). Design for it:

  1. Know where a subject's data lives (use lineage and a catalog tag for the identifier column).
  2. Minimize copies: the fewer places hold the data, the fewer deletions you need.
  3. Support row-level deletes in the lake. Table formats make this possible with MERGE or DELETE; plain Parquet files would need rewrites.
  4. Remember backups, logs and derived data (aggregates are usually fine, but ML training sets and caches may not be).
  5. Set retention periods and delete automatically when they expire (lifecycle rules, TTLs).
Python Process an erasure request across tables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the real world you may anonymize order rows (keep the amount, drop the link to the person) rather than delete them, if finance rules require keeping transactions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is removing the name column not enough to anonymize a dataset?

    Combinations of indirect attributes are often unique to one person.
  2. 2. Why use HMAC with a secret instead of a plain hash for email pseudonyms?

    A keyed hash blocks dictionary attacks on low-entropy values like emails.
  3. 3. What does least privilege mean?

    Minimal, role-based grants limit the damage of mistakes and breaches.
  4. 4. Pseudonymized data under GDPR is

    Pseudonymization reduces risk but does not remove the data from privacy law.

Summary

  • Classify columns, watch for quasi-identifiers, and find PII with scans plus owner review.
  • Mask, pseudonymize (keyed hash), tokenize, encrypt or generalize depending on the need.
  • Enforce least privilege with roles, column masking and row filters; audit access.
  • Plan retention and deletion from the start, and minimize copies.

Next: Cost Optimization and FinOps for Data