Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 47 ยท HDFS

NameNode HA, Federation and Operations

Learn how HDFS removes the NameNode single point of failure with standby nodes, journal nodes and ZooKeeper, plus federation, small files and the balancer.

The NameNode is the brain of HDFS. It keeps the whole namespace and the block map in memory, and every read and write starts with a question to it. In the first Hadoop versions there was exactly one NameNode: if it died, the entire cluster was unusable until someone restarted it. That is a single point of failure (SPOF), and it was the most famous weakness of early Hadoop.

This lesson explains how modern HDFS removes the SPOF with a standby NameNode, journal nodes and automatic failover, how federation spreads the namespace over several NameNodes, why millions of small files hurt, and what the balancer does. It continues from The HDFS Read and Write Path.

None of this can run on the site. The configuration is read-only. The runnable examples are stand-ins: a failover simulation and a memory estimate for the NameNode's metadata.

Why the NameNode is special

Recall from HDFS Architecture that the NameNode holds two things:

  • the namespace (directory tree, file names, permissions, which blocks make up each file), saved on disk as an fsimage snapshot plus an edit log of every change since;
  • the block map (which DataNodes hold each block), which is not saved but rebuilt from DataNode block reports.

Losing the namespace means losing the meaning of all the blocks. Even a clean restart is slow: load a large fsimage, replay edits, then wait for block reports. On a big cluster that can take tens of minutes to hours, all of it downtime.

High availability with a standby NameNode

HDFS High Availability (HA) runs two NameNodes: one active, serving all clients, and one standby, a hot spare that is ready to take over within seconds.

For a takeover to be fast, the standby must have nearly the same state as the active one. Two pieces make that work.

  1. Shared edit log in JournalNodes. The active NameNode writes each edit to a group of JournalNodes (a quorum, normally 3 or 5, small daemons that just store edits). A write counts as committed when a majority acknowledges it. The standby reads the edits as they arrive and applies them to its own in-memory namespace.
  2. Block reports to both. DataNodes send heartbeats and block reports to both NameNodes, so the standby also knows where all the blocks are. It periodically does checkpointing: merges the edits into a new fsimage (the job once done by the old "secondary NameNode") and sends it back.

HA in four parts

Active NameNode Serves every client request.
Standby NameNode Hot spare that replays edits.
JournalNodes Majority-stored shared edit log.
ZooKeeper Decides who is active.

Clients and DataNodes know both NameNodes.

1 / 2

Automatic failover and fencing

Each NameNode runs a small helper, the ZKFailoverController (ZKFC). ZKFC watches the health of its NameNode and holds a lock in ZooKeeper, a coordination service built for exactly this kind of agreement. Whoever holds the lock is active. If the active NameNode crashes or hangs, its ZooKeeper session expires, the lock is released, and the standby's ZKFC grabs it and promotes the standby.

The dangerous case is a hung old active that wakes up and still thinks it is in charge: two active NameNodes would corrupt the namespace (split brain). Two protections stop that:

  • JournalNodes only accept writes from the newest epoch number, so the old writer is rejected by the quorum.
  • Fencing methods (for example sshfence, or a custom script) forcibly stop the old active before the new one takes over.
flowchart LR
    ZK["ZooKeeper<br/>lock"] --- A["Active NN"]
    ZK --- S["Standby NN"]
    A -->|edits| J["JournalNodes<br/>(quorum of 3)"]
    J -->|edits| S
    D["DataNodes"] -->|reports| A
    D -->|reports| S

Both NameNodes get reports from DataNodes; only the active one writes edits.

A minimal HA configuration names a nameservice and the two NameNodes. Clients address the nameservice (hdfs://mycluster), never one specific host:

XML
<property><name>dfs.nameservices</name><value>mycluster</value></property>
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://jn1:8485;jn2:8485;jn3:8485/mycluster</value>
</property>
<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property>
<property><name>ha.zookeeper.quorum</name><value>zk1:2181,zk2:2181,zk3:2181</value></property>

The next stand-in models a quorum journal with three nodes and a failover with epochs, so that a stale old active cannot write.

Python Journal quorum and failover with epochs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Federation: more than one namespace

HA protects against failure but does not fix a second limit: one NameNode holds everything in one JVM's memory, so the number of files is capped by one machine's RAM, and all metadata operations pass through one process. HDFS Federation adds several independent NameNodes, each owning a part of the namespace (for example /user, /data, /tmp).

  • Each NameNode manages its own namespace volume and its own block pool (the blocks that belong to it).
  • DataNodes register with all NameNodes and store blocks of every pool, so the disks are shared.
  • NameNodes do not talk to each other. Clients see one tree through a ViewFs mount table that maps path prefixes to NameNodes.

Each federated NameNode is usually also an HA pair. Federation scales metadata and isolates teams, at the cost of more operational complexity, so most clusters use it only when one NameNode is truly at its limit.

The small-files problem

HDFS likes few large files. Every file, directory and block is an object in NameNode memory. A common rule of thumb is about 150 bytes per object, and each small file costs at least two objects: the file and its single block. Ten million tiny files can eat gigabytes of heap while holding almost no data. They also create ten million tiny map tasks in MapReduce, each with seconds of startup cost.

Note that a small file does not waste a whole 128 MB of disk: a 1 MB file uses 1 MB per replica. The cost is NameNode memory and job overhead, not disk.

Python NameNode memory for small files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the same 100 TB needs a vast amount of heap when files are 100 KB, but almost nothing when files are 512 MB. Fixes are all about merging: write bigger files in the first place, compact them with a periodic job, store many records in one container such as a SequenceFile, Avro or Parquet file (Partitioning, Bucketing and File Layout covers compaction), or use Hadoop Archives (HAR) for rarely used data.

Balancer and other operations

Over time some DataNodes fill up more than others, for example after adding new empty nodes. The balancer moves blocks from over-used to under-used nodes until every node's utilization is within a threshold of the cluster average, while keeping the placement rules (rack awareness) intact.

Terminal
hdfs balancer -threshold 10            # within 10 percentage points of the average
hdfs dfsadmin -setBalancerBandwidth 104857600   # limit it to 100 MB/s

Run it in quiet periods, because moving blocks uses network and disk. Other routine operations:

  • Decommissioning a DataNode: list it in the excludes file and run hdfs dfsadmin -refreshNodes. HDFS copies its blocks elsewhere before letting it go, so no data is lost.
  • Safe mode (hdfs dfsadmin -safemode get|enter|leave): a read-only state the NameNode uses at start-up.
  • Rolling upgrades with HA: upgrade the standby, fail over, then upgrade the former active, so users see almost no downtime.
  • Disk balancer (hdfs diskbalancer) evens out usage between the disks inside one DataNode.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does NameNode HA solve?

    With an active and standby NameNode, losing one NameNode no longer takes the whole file system down.
  2. 2. What are JournalNodes for?

    The active NameNode writes edits to a quorum of JournalNodes, and the standby reads them to stay current.
  3. 3. Why is a huge number of tiny files bad for HDFS?

    A small file uses only its real size on disk, but it still costs NameNode heap (file and block objects) and one map task.
  4. 4. What does federation add?

    Federation scales the namespace across NameNodes with separate block pools. Failover is HA, a different feature.

Summary

  • A single NameNode is a SPOF and its memory caps the number of files.
  • HA uses an active and a standby NameNode, a JournalNode quorum for edits, ZKFC plus ZooKeeper for failover, and fencing against split brain.
  • Federation gives several NameNodes their own namespace and block pool over shared DataNodes, joined by ViewFs.
  • Small files cost NameNode memory (about 150 bytes per object) and create many tiny tasks; merge or compact them.
  • The balancer evens out DataNode usage; decommissioning and rolling upgrades keep maintenance safe.

Next: YARN: Resources and Scheduling.