Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Getting Started

HBase and the Hadoop Ecosystem

See how HBase uses HDFS for storage and ZooKeeper for coordination, how it relates to MapReduce, Spark, Hive and Phoenix, and what each component is for.

HBase is never alone. A running cluster is a small team: a file system that keeps the bytes safe, a coordination service that keeps everyone in agreement, and HBase itself that organizes the bytes into tables. Add tools for analytics and SQL and you have what people call "the Hadoop ecosystem". This lesson gives you the map, so the architecture lessons in the next chapter make sense.

The layers

The HBase stack

Your application Java client, REST, Thrift
HBase Master and RegionServers
ZooKeeper Leader election and liveness
HDFS Replicated files on many disks

Each layer does one job.

1 / 2

HDFS: the durable storage

The Hadoop Distributed File System splits files into large blocks (128 MB by default) and stores each block on several DataNodes. A NameNode keeps the map of which blocks make up which file. If a disk dies, the NameNode notices missing replicas and copies them elsewhere.

HBase relies on this. It writes two kinds of files to HDFS:

  • HFiles, the immutable, sorted data files that hold table contents.
  • Write-ahead log (WAL) files, a journal of recent changes used to recover after a crash.

Because HDFS replicates, a RegionServer machine can die without losing data: another RegionServer simply opens the same files from HDFS and takes over. This is why HBase servers are "nearly stateless" and can be replaced quickly. HDFS files are write-once, so HBase never edits a file in place; it writes new ones and merges old ones in the background (compaction, covered in The Write Path).

ZooKeeper: the coordinator

Apache ZooKeeper is a small, strongly consistent service holding tiny pieces of data in a tree of "znodes". HBase uses it for:

  • Master election. Several Masters may run; the one that creates the master znode is active, the others wait as backups.
  • Liveness tracking. Each RegionServer keeps an ephemeral znode that disappears when the server stops responding, so the Master learns about failures quickly.
  • Locating the meta table. Clients ask ZooKeeper where the hbase:meta region lives (more in Regions, Splits and the Meta Table).
  • Cluster-wide flags, such as whether the balancer is on.

A production ZooKeeper runs as an odd-sized quorum (3 or 5 servers) so that a majority can still agree if one fails. Notice that ZooKeeper does not carry your data; it only carries coordination information, which is why it stays small.

MapReduce and Spark

HBase tables can be inputs and outputs of batch jobs. TableInputFormat turns each region into a split, so a job reads regions in parallel (and tries to run tasks near the data). Spark connectors do the same. Typical uses: building a daily report from raw events, bulk loading a table (writing HFiles directly, skipping the normal write path), or exporting a snapshot.

Hive and Phoenix: SQL on top

Writing Scan objects by hand is tedious for analysts. Two common ways to add SQL:

  • Apache Hive can map an HBase table to an external table and run SQL over it. It is good for batch analytics but slow for interactive lookups, because it compiles queries to distributed jobs.
  • Apache Phoenix is a SQL layer implemented with HBase coprocessors. It gives you CREATE TABLE, SELECT, joins, secondary indexes and low-latency queries, translating them into scans and gets. If you want a SQL-first experience on HBase, Phoenix is the usual choice.

Access paths

Besides the native Java client, HBase has a REST gateway (hbase rest start), a Thrift server for languages like Python or Go, and a shell built on JRuby. Non-Java clients speak to a gateway, which uses the Java client internally.

Modeling the dependencies

Here is a Java stand-in showing why the stack is layered. It models a "cluster" where ZooKeeper tracks live servers and HDFS keeps data. When a RegionServer dies, its regions are reassigned and the data is still there, because the data was never on the server alone.

Java Stand-in: a server dies, the data survives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Choosing your distribution and versions

You can run the Apache release yourself, use a vendor distribution, or rent a managed service from a cloud provider. Whatever you choose, the compatible versions matter: an HBase 2.x release is built against a Hadoop 2.10 or 3.x line and a ZooKeeper 3.x. Check the "Hadoop version support" matrix of your release before upgrading any piece.

A second small stand-in: a version compatibility table you might keep next to your runbook. The numbers here are examples to show the technique, not a real support matrix.

Java Stand-in: a compatibility lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What kind of files does HBase store in HDFS?

    HFiles hold table data and WAL files hold recent changes for crash recovery.
  2. 2. Why does ZooKeeper run as 3 or 5 servers?

    ZooKeeper needs a majority (a quorum) to make decisions, so an odd number of servers tolerates failures.
  3. 3. Which tool gives HBase SQL with secondary indexes and low-latency queries?

    Phoenix is a SQL layer built on HBase coprocessors. Hive can query HBase too but targets batch analytics.
  4. 4. A RegionServer machine crashes. Why is the table data not lost?

    Data lives in replicated HDFS files, not on the RegionServer's own disk alone.

Summary

  • HDFS stores HFiles and WAL files with replication; HBase servers hold no irreplaceable local state.
  • ZooKeeper elects the active Master, tracks live servers and points clients to the meta region.
  • MapReduce and Spark read and write tables in parallel, one split per region.
  • Hive and Phoenix add SQL; REST and Thrift gateways serve non-Java clients.
  • Check version compatibility between HBase, Hadoop, ZooKeeper and Java before upgrading.

Next: Tables, Rows, Column Families and Qualifiers