Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 47 ยท Architecture and Operations

ZooKeeper for Coordination

Learn what ZooKeeper does in Hadoop: znodes, ensembles and quorum, sessions, watches, leader election, and its role in HDFS and YARN high availability and HBase.

Distributed systems need a small, reliable place to answer questions such as: Which NameNode is active right now? Which server is the leader? What is the current configuration? Is that worker still alive? Letting every node decide for itself leads to split-brain (two nodes both believing they are the leader) and corrupted data. Apache ZooKeeper is a coordination service built to answer such questions consistently. It is not a database for your data: it stores tiny pieces of control information, but it stores them with strong guarantees.

This lesson explains ZooKeeper's data model (znodes), its ensemble and quorum, sessions and watches, the leader election recipe, and where it appears in Hadoop. It builds on Distributed Systems in Plain Words. ZooKeeper cannot run on this site, so the shell commands are read-only; the runnable example is a Python stand-in with znodes, ephemeral nodes and watches.

The data model: znodes

ZooKeeper looks like a small file system. Data lives in a tree of znodes, addressed by paths such as /hadoop-ha/mycluster/ActiveStandbyElectorLock. Each znode holds a small amount of data (by default under 1 MB; usually bytes to a few kilobytes) and can also have children. Unlike a normal file system, every znode can hold data and children.

Znodes come in different kinds:

  • Persistent: stays until someone deletes it. Good for configuration.
  • Ephemeral: tied to the client session; when the client disconnects or dies, ZooKeeper deletes the znode automatically. Good for "I am alive" markers and locks. Ephemeral znodes cannot have children.
  • Sequential: ZooKeeper appends a growing, unique number to the name (/election/n_0000000007). Combined with ephemeral, this gives each participant a place in line.

Every znode keeps a version counter, so updates can be conditional ("set this value only if the version is still 5"), which prevents lost updates between competing clients.

Terminal
zkCli.sh -server localhost:2181
create /app "config-v1"                    # persistent znode
create -e /app/worker1 "10.0.0.5"          # ephemeral: disappears when this session ends
create -s /app/job- "x"                    # sequential: becomes /app/job-0000000000
ls /app
get /app
set /app "config-v2"
delete /app/worker1

Ensemble, quorum and consistency

A single ZooKeeper server would be a single point of failure, so ZooKeeper runs as an ensemble of an odd number of servers, usually 3 or 5. One is the leader; the others are followers. Writes go through the leader, which broadcasts them with an atomic broadcast protocol (ZAB). A write succeeds once a majority (quorum) of servers have it. Reads can be served by any server, so they scale with the number of servers (a follower may lag slightly behind; a client can call sync first if it must see the latest write).

Why odd numbers? A quorum is a majority: 2 of 3, 3 of 5. A 3-server ensemble tolerates 1 failure; a 5-server ensemble tolerates 2. A 4-server ensemble also tolerates only 1 failure (it still needs 3), so the fourth server adds cost without extra safety. If the ensemble loses its majority, ZooKeeper stops accepting writes rather than risk inconsistency, which is the correct choice for coordination. Each server's zoo.cfg lists the peers:

INI
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=10
syncLimit=5
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888

ZooKeeper keeps its whole data set in memory and logs changes to disk, which is why it is fast but must stay small. Give it fast disks for the transaction log and keep it off busy shared machines, because slow ZooKeeper means slow failover in everything that depends on it (see Sizing, Monitoring and Running a Cluster).

Sessions and watches

A client connects to one server and opens a session with a timeout. The client sends regular heartbeats; if the servers hear nothing for the timeout, the session expires and all its ephemeral znodes disappear. This is how a crashed process is noticed.

A watch asks ZooKeeper to notify the client when a znode changes (created, deleted, data changed, children changed). Watches are one-time triggers: once fired, you must set the watch again, and you should re-read the znode when you do, because it may have changed again in between. Instead of polling every second, a client waits for the notification.

Python A mini ZooKeeper: ephemeral nodes and watches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Leader election

The standard recipe uses ephemeral sequential znodes. Every candidate creates /election/n_ with the ephemeral and sequential flags and gets a number. The candidate with the lowest number is the leader. Every other candidate watches only the znode just before its own (not the leader, which would wake everyone up at once, the "herd effect"). When the leader's session dies, its znode vanishes, the next candidate's watch fires, it checks whether it is now the lowest and, if so, becomes leader. The same pattern builds distributed locks and queues.

Python Leader election with ephemeral sequential nodes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where ZooKeeper appears in Hadoop

  • HDFS automatic failover. Each NameNode runs a ZKFailoverController (ZKFC). The active NameNode's ZKFC holds an ephemeral lock znode; if the NameNode or its host dies, the lock disappears, the standby's ZKFC wins the election and promotes its NameNode (see NameNode HA, Federation and Operations). ZooKeeper only decides who is active; the journal nodes and fencing protect the data.
  • YARN ResourceManager HA. Active/standby ResourceManagers elect through ZooKeeper, which can also store application state for recovery (see YARN: Resources and Scheduling).
  • HBase. Masters and region servers register ephemeral znodes; clients ask ZooKeeper where the hbase:meta table lives (see HBase: Wide-Column Storage).
  • Other tools. Hive uses it for table locks and HiveServer2 discovery; Solr, older Kafka versions (newer Kafka replaced it with its own KRaft quorum) and many other systems rely on it too.

Good practice

Keep znodes small and few; do not store application data in ZooKeeper. Run 3 or 5 servers on separate machines, ideally in separate failure domains, with the transaction log on its own fast disk. Watch session timeouts: a garbage-collection pause in a client can exceed the timeout and make ZooKeeper believe the client died, so tune heap and timeouts together. Secure it (Kerberos or SASL and ACLs on znodes), because whoever can write the election znode can steer a cluster (see Security, Governance and Data Quality).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many server failures can a 5-server ZooKeeper ensemble tolerate?

    A quorum needs a majority, which is 3 of 5. So up to 2 servers can fail.
  2. 2. What happens to an ephemeral znode when its client session ends?

    Ephemeral znodes exist only as long as the session that created them, which makes them ideal for presence markers and locks.
  3. 3. In the leader-election recipe, which candidate is the leader?

    Candidates create ephemeral sequential znodes; the lowest number leads, and each other candidate watches the node just before its own.
  4. 4. What does ZooKeeper do for HDFS high availability?

    ZooKeeper only coordinates who is active; the data and edit log are handled by HDFS and its journal nodes.

Summary

  • ZooKeeper is a small, strongly consistent coordination service holding tiny pieces of control data in a tree of znodes.
  • Znodes can be persistent, ephemeral (die with the session) and sequential; versions allow safe conditional updates.
  • An ensemble of an odd number of servers (3 or 5) needs a majority to accept writes, so 2f + 1 servers tolerate f failures.
  • Sessions and heartbeats detect dead clients; one-time watches notify clients of changes.
  • Leader election and locks use ephemeral sequential znodes; HDFS and YARN HA, HBase and Hive depend on it.

Next: see how clusters are sized and run in Sizing, Monitoring and Running a Cluster.