Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 79 ยท Distributed Databases, NoSQL and Analytics

HBase, HDFS and Hive Overview

Understand the Hadoop data stack from a DBMS view: HDFS storage, HBase regions, HMaster, RegionServers and ZooKeeper, Hive SQL on files, and when HBase beats an RDBMS.

When data reaches hundreds of terabytes, the single-server designs of the earlier lessons stop working, and even a sharded relational database becomes expensive to run. The Hadoop ecosystem was built for that scale. It runs on cheap commodity machines, assumes that some of them will fail every day, and spreads both data and work across the cluster. Seen as a DBMS person, three of its parts matter most:

  • HDFS, a distributed file system that stores the bytes.
  • HBase, a wide-column NoSQL database that gives random reads and writes on top of HDFS.
  • Hive, a data warehouse layer that lets you run SQL over files in HDFS.

This lesson is an overview that connects them to the ideas you already know: NoSQL families, partitioning, LSM trees and warehouses. The Hadoop and Big Data tutorial goes much deeper into each piece.

The Hadoop database stack

HDFS Big files split into replicated blocks.
HBase Random read and write by row key.
Hive SQL queries over files.

Each layer solves a different problem.

1 / 2

HDFS in one page

The Hadoop Distributed File System stores very large files, usually written once and read many times. A file is cut into big blocks (128 MB by default) and each block is replicated on several machines (3 copies by default), so losing a disk or a whole server loses no data.

  • The NameNode keeps the metadata: the directory tree, which blocks make up each file and which DataNodes hold each block. It is the file system's catalog.
  • The DataNodes store the blocks and report to the NameNode with periodic heartbeats.

HDFS is built for large sequential reads and appends. It is a poor fit for lots of tiny files, for low-latency random access, or for updating a file in place. Those weaknesses are exactly what HBase and Hive work around. See HDFS architecture for the details.

HBase: a wide-column store on HDFS

HBase is an open-source, distributed, versioned, wide-column database modeled on Google's Bigtable. It stores its data files in HDFS and so inherits replication and fault tolerance. It is designed for tables with billions of rows and millions of columns, where an application needs fast random reads and writes by key: messaging systems, time-series data, user profiles, sensor feeds.

The data model

An HBase table is a sorted map. The pieces are:

  • Row key: the primary key. Rows are stored sorted by row key (as raw bytes, in lexicographic order), so a range scan over adjacent keys is cheap. Row design is the most important design decision.
  • Column family: a group of columns stored together on disk. Families are declared when the table is created and should be few (one to three is typical).
  • Column qualifier: the column name inside a family, written family:qualifier. Qualifiers are not declared in advance; each row can have different ones, which is why the table is "wide" and sparse. A missing cell costs nothing.
  • Cell and version: a cell is identified by (row, family, qualifier, timestamp). HBase keeps several timestamped versions of each cell, newest first.

The Python model below is a small teaching imitation of this structure: sorted keys, fixed families, free qualifiers, versions and range scans.

Python A tiny HBase-style table: families, versions, sorted scans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rows 1 and 2 have different columns (info:city against stats:logins), which a relational table could only model with many NULLs. The scan returns the half-open key range in sorted order and stops before user#0003. Adding a cell to an undeclared family fails, because families are part of the table definition.

Regions, RegionServers, HMaster and ZooKeeper

HBase scales by cutting each table into regions. A region holds a contiguous range of row keys, and all regions of a table together cover the whole key space. When a region grows past a size limit, it splits into two. This is automatic range partitioning (see partitioning). The cluster has these roles:

  • A RegionServer is a worker process that serves reads and writes for the regions assigned to it. A server typically hosts tens to hundreds of regions. It keeps recent writes in memory (the MemStore) and in a write-ahead log, and flushes them as sorted HFiles to HDFS, the LSM-tree write path.
  • The HMaster is the coordinator. It assigns regions to RegionServers, rebalances load, reassigns regions of a dead server and executes administrative operations such as creating or altering tables. It is not on the read/write path: clients talk to RegionServers directly, so a brief master outage does not stop normal traffic.
  • ZooKeeper is a small, replicated coordination service. HBase uses it to elect the active HMaster, to detect which RegionServers are alive, and to store the location of the special catalog table hbase:meta. That table maps row-key ranges to regions and their servers.

A client that wants row user#0006 asks ZooKeeper where hbase:meta lives, looks the key up there to find the region and its server, caches the answer, and from then on talks to that RegionServer directly. The next simulation shows regions splitting as rows arrive and a key being routed to its region.

Python Region splits and key lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three splits happen as the twelve rows arrive, and the new regions are spread over the three servers. The final line is the client's routing step: the greatest region start key not above user#0006 is user#0005, which rs2 serves.

One warning follows from range partitioning: if row keys grow monotonically (timestamps, auto-increment ids), every new write lands in the last region and one RegionServer takes all the traffic (a hotspot). Designers avoid it by salting or hashing a prefix onto the key, or by putting a high-cardinality field first.

Consistency and queries

HBase offers strong consistency for single-row operations: all reads and writes of one row go to one RegionServer, and a row update is atomic across its column families. There are no multi-row transactions and no joins in the core API; the operations are Get, Put, Scan and Delete. In terms of the CAP theorem, HBase favors consistency over availability for a region while it is being recovered. SQL is available as an extra layer (for example Apache Phoenix), but the native model is the key-based one above.

RDBMS compared with HBase

RDBMS HBase
Model Tables with fixed columns, relationships Sorted map of row key to families, qualifiers and versions
Schema Rigid, declared up front Only table and column families; qualifiers are free
Query language SQL with joins and aggregates Get, Put, Scan; no joins (SQL via add-ons)
Transactions Multi-row ACID Atomic per row only
Scaling Mostly vertical; sharding is manual Horizontal by automatic region splits
Indexes Many secondary indexes Row key only (secondary indexes need extra design)
Best for Complex relational data, moderate size Huge sparse tables with key-based access
Storage Local disks HDFS, replicated

Choose HBase when the data is enormous, access is mostly by key or key range, writes are heavy, and rows have a variable set of columns. Choose a relational database when you need joins, ad-hoc queries and multi-row transactions.

Hive: SQL on top of files

HBase serves applications that need low-latency access. Apache Hive serves analysts who want to query huge data sets with SQL. Hive is a data warehouse system built on Hadoop:

  • Data stays as ordinary files in HDFS (text, CSV, ORC, Parquet).
  • The metastore (a small relational database) holds the table definitions: names, columns, types, file locations and partitions.
  • Queries are written in HiveQL, which is close to SQL. Hive compiles them into distributed jobs (MapReduce originally, now usually Tez or Spark) that run across the cluster.

Hive uses schema on read. A relational database checks types when data is written; Hive only describes how to interpret the files, and the interpretation is applied when the data is read, so values that do not fit usually become NULL. Large tables are partitioned: each value of a partition column (for example the date) is a separate HDFS directory, and a query that filters on it reads only the matching directories (partition pruning). The example imitates both ideas with plain SQL.

SQL Schema on read and partition filters, Hive style

Press Run (or Ctrl + Enter) to see the result here.

The bad value n/a was stored without complaint and became NULL on read, so docs has three views but only two valid times. This is flexible for messy log data and risky if you never check quality. In real HiveQL you declare the table like this (read-only illustration, since it needs a cluster):

SQL
-- HiveQL: read-only illustration
CREATE EXTERNAL TABLE page_views (page STRING, visitor STRING, millis INT)
PARTITIONED BY (view_date STRING)
STORED AS PARQUET
LOCATION '/data/page_views';

SELECT page, AVG(millis) FROM page_views WHERE view_date = '2025-03-02' GROUP BY page;

Hive queries take seconds to minutes, not milliseconds, and Hive tables are not meant for row-by-row transactional updates. HBase and Hive are therefore complementary: HBase serves the live application by key, Hive (or Spark SQL) analyzes the history. The Hive basics lesson and HBase concepts lesson in the Hadoop tutorial continue from here.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the role of the HMaster in HBase?

    Clients read and write through RegionServers directly. The HMaster coordinates, assigning regions, reacting to server failures and running schema changes.
  2. 2. What is an HBase region?

    A table is divided into regions by row-key range, and a region splits automatically when it becomes too large.
  3. 3. Why does a monotonically increasing row key such as a timestamp cause problems?

    Because rows are sorted by key and regions are key ranges, sequential keys always fall into the newest region. Salting or hashing a prefix spreads the load.
  4. 4. What does "schema on read" mean in Hive?

    Hive stores the table definition in the metastore and interprets the underlying files at query time, unlike an RDBMS that validates data on insert.
  5. 5. Which workload suits HBase better than a relational database?

    HBase scales horizontally for key-based access to huge sparse tables. It lacks joins and multi-row transactions.

Summary

  • HDFS stores files as large replicated blocks, with a NameNode for metadata and DataNodes for data; it suits big sequential access, not small random updates.
  • HBase is a wide-column store on HDFS: tables are sorted maps of row key, column families, free-form qualifiers and timestamped versions.
  • A table is split into key-range regions; RegionServers serve them, the HMaster assigns and rebalances them, and ZooKeeper tracks live servers, the active master and the location of hbase:meta.
  • HBase gives single-row atomicity and key-based access, with no joins or multi-row transactions; sequential row keys create hotspots.
  • Compared with an RDBMS it trades SQL, joins and transactions for horizontal scaling and a flexible schema.
  • Hive provides SQL (HiveQL) over files in HDFS with a metastore, schema on read and partition pruning, for batch analytics rather than low-latency access.

Next: OLTP vs OLAP, Columnar Storage and Data Warehouses.