Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 47 ยท Architecture and Operations

Security, Governance and Data Quality

Learn how Hadoop clusters are secured with Kerberos, Ranger and encryption, how lineage and catalogs help, and how to check data quality with rules.

A cluster that holds years of customer data is a very attractive target, and it is also a legal liability. Out of the box, early Hadoop trusted whatever user name a client claimed. Modern deployments add layers: proving who you are, deciding what you may touch, protecting bytes on disk and on the wire, and keeping a record of where data came from. On top of that comes a quieter problem: data that is allowed but wrong.

This lesson walks through those layers, one by one: authentication, authorization, encryption, governance (catalogs and lineage), privacy deletes, and data quality. The security tools themselves cannot run on this site, so their commands are read-only. The runnable examples are Python stand-ins that show the idea: a rule checker, a pseudonymizer and a "forget this person" job.

Kerberos, Ranger, KMS and the catalogs are real products that need servers. Every code block marked read-only below is real configuration or a real command. The TryIt examples are small simulations: they show the logic, not the products.

The layers of protection

Four questions of security

Authentication Who are you really?
Authorization What may you read or change?
Encryption Are the bytes unreadable to thieves?
Auditing Who did what, and when?
1 / 2

Authentication with Kerberos

By default, a Hadoop client says "I am alice" and the cluster believes it. In "simple" mode, anyone who can reach the NameNode can pretend to be the HDFS superuser. Production clusters switch on Kerberos, a ticket-based protocol in which a trusted server, the KDC (Key Distribution Center), vouches for each user and each service.

In short: you log in once and receive a ticket-granting ticket. When you talk to a service such as the NameNode, you present a short-lived service ticket issued by the KDC. Passwords never travel to the service. Every daemon (NameNode, DataNode, ResourceManager) has its own principal (for example nn/[email protected]) and a keytab file holding its key.

The switch is a few lines in core-site.xml (read-only):

XML
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>
<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

And from the user's side:

Terminal
kinit [email protected]      # asks for the password, stores a ticket
klist                        # shows the ticket and its expiry
hdfs dfs -ls /data/sales     # now works with that identity

Long-running jobs outlive a ticket, so Hadoop uses delegation tokens: the job obtains tokens at submit time and renews them, so tasks on hundreds of nodes never contact the KDC themselves. Web UIs and the REST gateways typically use Kerberos through SPNEGO, or a gateway such as Apache Knox that fronts the cluster with LDAP or single sign-on.

Authorization: permissions, ACLs and Ranger

HDFS has POSIX-style permissions (owner, group, others; see Working with HDFS from the Shell) plus ACLs for finer rules. That is enough for file access but not for column-level questions such as "analysts may query the orders table but must not see card_number".

Apache Ranger adds central, policy-based authorization for many services (HDFS, Hive, HBase, Kafka, YARN queues). You define a policy once in the Ranger web UI, and each service runs a small plugin that downloads the policies and checks every request locally. Ranger can also apply column masking (show XXXX-XXXX-XXXX-1234) and row filters (a regional manager sees only rows of their region), and it records an audit trail of allowed and denied requests. A similar role is played by Apache Sentry in older distributions, and by cloud IAM policies on object stores.

A good policy follows least privilege: groups, not individual users; read access by default; write access only for pipelines.

Encryption: at rest and in transit

At rest. HDFS transparent encryption splits the cluster into encryption zones, which are directories whose files are encrypted. Each file gets its own data encryption key (DEK). The DEK is itself encrypted with a zone key held in a Key Management Server (KMS). The NameNode and DataNodes only ever see the encrypted DEK, so even an HDFS administrator cannot read the data without KMS permission.

Terminal
hadoop key create sales-key
hdfs crypto -createZone -keyName sales-key -path /data/sales
hdfs crypto -listZones

Disk-level encryption of the nodes protects stolen drives but not a misbehaving admin; HDFS zones add separation of duties.

In transit. Three channels need protecting: RPC calls (set hadoop.rpc.protection to privacy), the block data transfer between clients and DataNodes (dfs.encrypt.data.transfer = true), and the HTTP interfaces and MapReduce shuffle (TLS via dfs.http.policy = HTTPS_ONLY and the shuffle's SSL setting). Encryption in transit costs CPU, which is one reason it is configured deliberately rather than assumed.

Governance: catalogs and lineage

A lake with ten thousand tables is useless if nobody knows what they mean. A data catalog (Apache Atlas in the Hadoop world; DataHub, Amundsen or cloud catalogs elsewhere) stores metadata: dataset names, owners, descriptions, schemas, tags such as PII or finance, and quality scores. It lets analysts search for data and lets Ranger apply tag-based policies ("mask everything tagged PII").

Lineage records how data flows: this report was built from that Hive table, which was loaded by that job from those raw files. Hooks in Hive, Spark and Sqoop report each job's inputs and outputs to the catalog, which draws the graph.

flowchart LR
    A["raw/clicks"] --> B["job: clean_clicks"]
    B --> C["curated/clicks"]
    C --> D["job: daily_summary"]
    E["curated/users"] --> D
    D --> F["report: daily_kpis"]

Lineage answers two questions: "where did this number come from?" and "what breaks if I change this table?"

Lineage is also a compliance tool. When a regulator asks which systems hold a customer's email address, you want a graph to follow, not a week of detective work.

Privacy: deleting a person from an immutable lake

Laws such as the GDPR give people a right to erasure. That is awkward on HDFS, where files are write-once and a person's rows sit inside large Parquet files next to millions of others. There is no DELETE FROM for a plain file. The usual solutions are:

  • Rewrite the affected files without those rows (a periodic batch job, since GDPR allows a reasonable delay).
  • Table formats with row-level deletes such as Delta, Iceberg or Hive ACID tables (see Data Lakes, Warehouses and Lakehouses).
  • Pseudonymization: store a random or hashed ID in the lake and keep the mapping from ID to real person in one small, well-protected table. Deleting that mapping row makes the lake data anonymous in practice.
  • Crypto-shredding: encrypt each person's data with their own key and delete the key.

Here is a stand-in that pseudonymizes email addresses with a salted hash and then "forgets" one user by dropping the mapping.

Python Pseudonymize, then forget one person

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Be honest about the limits: a hash of an email can be reversed by anyone who can guess emails and knows the salt, so real systems keep the salt secret or use random IDs. Pseudonymized data may still count as personal data under some laws; ask your privacy team.

Data quality: trust, but verify

Security decides who may read the data. Data quality decides whether anyone should. Bad data arrives constantly: a source system renames a column, a mobile app sends prices as strings, a job runs twice and duplicates a day. Because pipelines are automated, the bad rows flow into reports before anyone notices.

The fix is to turn expectations into rules that run with every load:

  • Completeness: required fields are not null.
  • Validity: values are in range or match a pattern (an email has an @, a rating is 1 to 5).
  • Uniqueness: keys do not repeat.
  • Consistency: related fields agree (shipping date is not before the order date).
  • Freshness and volume: yesterday's partition exists and has roughly the usual row count.

Tools such as Deequ, Great Expectations and Soda run these checks on Spark or SQL. A failed rule can stop the pipeline (quarantine the batch) or just raise an alert. Here is a tiny stand-in that applies the same ideas.

Python A small data-quality rule checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the shape: rules are data (a name plus a function), so adding one is a single line, and the report names the exact bad rows. That is how the real tools work too, with a SQL or Spark engine doing the counting.

Putting it together

A realistic checklist for a new dataset: it lives in an encryption zone if sensitive; Ranger grants a group read access, with masking on tagged columns; the catalog lists owner, description and tags; lineage is captured automatically by the jobs that write it; quality rules run on every partition; and a documented process removes a person's rows on request.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does Kerberos solve in a Hadoop cluster?

    Kerberos is an authentication protocol: a KDC issues tickets so users and daemons can prove who they are. Authorization and encryption are separate layers.
  2. 2. Which tool lets you define central policies, column masks and audit logs across HDFS, Hive and HBase?

    Ranger plugins in each service fetch policies from a central admin and log every decision. The other tools handle data balancing, coordination and copying.
  3. 3. How does HDFS transparent encryption keep an HDFS administrator from reading the data?

    Each file's key is wrapped by a zone key that only the KMS can unwrap. Without KMS permission, the stored bytes are unreadable.
  4. 4. Why is deleting one person's rows from a Parquet-based lake hard, and what is one common solution?

    Rows are embedded in large immutable files. Batch rewrites, Delta/Iceberg/Hive ACID deletes, pseudonymization and crypto-shredding are the usual answers.
  5. 5. A nightly load suddenly has half the usual number of rows and no error. Which kind of data-quality rule would catch it?

    Volume and freshness rules compare today's batch with expectations, catching silent failures that no exception reports.

Summary

  • Secure clusters use Kerberos for authentication; Hadoop's default mode trusts claimed user names.
  • Ranger gives policy-based authorization, masking, row filters and audit trails; least privilege is the rule.
  • Encrypt at rest with HDFS encryption zones and a KMS, and in transit with RPC, data-transfer and TLS settings.
  • Catalogs make data findable; lineage shows where it came from and what depends on it.
  • Erasure requests need rewrites, row-level table formats, pseudonymization or crypto-shredding.
  • Data-quality rules (completeness, validity, uniqueness, consistency, freshness) run on every load and can stop bad batches.

Next: Sizing, Monitoring and Running a Cluster.