Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท Integration and Patterns

HBase with MapReduce, Spark and Hive

Run analytics over HBase: TableMapper and TableReducer, Spark connectors, snapshot scans and Hive external tables, with runnable Java models of the data flow.

HBase is built for fast random reads and writes by key, not for ad-hoc analytics. For reports, joins and machine-learning pipelines you pair it with batch engines: MapReduce, Apache Spark and Apache Hive. This lesson shows the three integration styles and the rule that decides how safe they are for your live traffic: whether the job reads through the region servers or reads HFile snapshots directly.

HBase, Hadoop, Spark and Hive cannot run in the live editors on this site. Their real code and configuration are shown in read-only blocks. The runnable Java blocks are labelled models: they reproduce how a table is split into input splits and how a Hive column mapping reads a row.

How a table becomes a batch job

Every engine uses the same idea: one input split per region. A job over a table of 40 regions has 40 tasks, each scanning the key range of one region, ideally on the machine that hosts it (data locality). Results are merged afterwards. This is the same pattern you saw in endpoint coprocessors, except the code runs in separate processes rather than inside the region server.

flowchart TB
  T["HBase table, 3 regions"] --> S1["Split 1 (region 1)"]
  T --> S2["Split 2 (region 2)"]
  T --> S3["Split 3 (region 3)"]
  S1 --> M1["Map task"]
  S2 --> M2["Map task"]
  S3 --> M3["Map task"]
  M1 --> R["Reduce / shuffle"]
  M2 --> R
  M3 --> R
  R --> O["Output (HDFS, HBase, ...)"]

Each map task scans one region. Region count controls parallelism.

MapReduce

HBase ships classes in org.apache.hadoop.hbase.mapreduce:

  • TableInputFormat, which creates one split per region (so a table with 3 regions runs 3 mappers).
  • TableMapper, a mapper that receives each row as a Result.
  • TableReducer, a reducer that emits Put or Delete back to a table through TableOutputFormat.
  • TableMapReduceUtil, helper to configure a job.

A read-only job that counts events per status:

Java
public class StatusCount {
    static class M extends TableMapper<Text, LongWritable> {
        private static final LongWritable ONE = new LongWritable(1);
        @Override
        protected void map(ImmutableBytesWritable row, Result r, Context ctx) throws IOException, InterruptedException {
            byte[] status = r.getValue(Bytes.toBytes("d"), Bytes.toBytes("status"));
            if (status != null) ctx.write(new Text(Bytes.toString(status)), ONE);
        }
    }

    static class R extends TableReducer<Text, LongWritable, ImmutableBytesWritable> {
        @Override
        protected void reduce(Text key, Iterable<LongWritable> vals, Context ctx) throws IOException, InterruptedException {
            long n = 0; for (LongWritable v : vals) n += v.get();
            Put put = new Put(Bytes.toBytes(key.toString()));
            put.addColumn(Bytes.toBytes("c"), Bytes.toBytes("n"), Bytes.toBytes(n));
            ctx.write(null, put);
        }
    }

    public static void main(String[] args) throws Exception {
        Job job = Job.getInstance(HBaseConfiguration.create(), "status-count");
        Scan scan = new Scan().addColumn(Bytes.toBytes("d"), Bytes.toBytes("status"));
        scan.setCaching(500);
        scan.setCacheBlocks(false);                      // do not pollute the block cache
        TableMapReduceUtil.initTableMapperJob("events", scan, M.class, Text.class, LongWritable.class, job);
        TableMapReduceUtil.initTableReducerJob("status_counts", R.class, job);
        job.waitForCompletion(true);
    }
}

Note setCacheBlocks(false): batch scans should never evict the hot working set (see block cache).

Java Map and reduce per region (model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Reading through the region servers or from snapshots

A scan-based job uses the normal read path: it consumes region server CPU, memory, block cache and I/O. A long job on a busy cluster slows your online users. Two options:

  1. Throttle and tune: setCaching, setCacheBlocks(false), run off-peak, use a dedicated cluster fed by replication.
  2. Read a snapshot: TableSnapshotInputFormat reads the snapshot's HFiles directly from HDFS and bypasses the region servers. The load on HBase is nearly zero. The data is only as fresh as the snapshot.
Java
TableMapReduceUtil.initTableSnapshotMapperJob("events_2026_11_12", scan, M.class, Text.class,
    LongWritable.class, job, true, new Path("/tmp/snapshot-restore"));

Spark

Spark reads HBase in several ways:

  • Hadoop input format: sc.newAPIHadoopRDD with TableInputFormat, giving an RDD of (ImmutableBytesWritable, Result). Works everywhere but you decode bytes yourself.
  • HBase Spark connector (hbase-connectors, hbase-spark): a DataFrame API with a catalog that maps columns to families and qualifiers, with predicate and column pushdown for row-key filters.
  • Phoenix-Spark (Thrift, REST and Phoenix): read tables with SQL types through Phoenix.

A catalog for the connector (read-only):

SCALA
val catalog = """{
  "table": {"namespace": "default", "name": "events"},
  "rowkey": "key",
  "columns": {
    "id":     {"cf": "rowkey", "col": "key",    "type": "string"},
    "status": {"cf": "d",      "col": "status", "type": "string"},
    "ms":     {"cf": "d",      "col": "ms",     "type": "long"}
  }
}"""
val df = spark.read.options(Map(HBaseTableCatalog.tableCatalog -> catalog))
  .format("org.apache.hadoop.hbase.spark").load()
df.filter($"status" === "error").groupBy("status").count().show()

Spark tasks are per region by default as well, so the same hotspot and parallelism rules apply. For big historical scans, prefer a snapshot-based read.

Hive

Hive can expose an HBase table as an external table using the HBaseStorageHandler. You declare how Hive columns map to row key and family:qualifier pairs, then query it with SQL:

SQL
CREATE EXTERNAL TABLE hbase_events (
  key    string,
  status string,
  ms     bigint
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,d:status,d:ms")
TBLPROPERTIES ("hbase.table.name" = "events");

SELECT status, COUNT(*) FROM hbase_events GROUP BY status;

The mapping string ":key,d:status,d:ms" lists, in order, one HBase location per Hive column: :key is the row key, the others are family:qualifier. Hive pushes simple row-key predicates down to HBase but otherwise scans, so it is for analysis, not low-latency serving.

Java Reading a Hive column mapping (model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Choosing a tool

Need Tool
Existing Hadoop jobs, simple counts, writing back to HBase MapReduce
Complex pipelines, DataFrames, ML Spark
SQL analysts on HBase data Hive, or Phoenix for low-latency SQL
Heavy scans without hurting production Any of them over a snapshot

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many input splits does TableInputFormat create?

    Each region becomes one split, so region count controls job parallelism.
  2. 2. Why use TableSnapshotInputFormat?

    A snapshot read bypasses the region servers, so analytics do not affect online traffic.
  3. 3. What should a batch scan set to protect the block cache?

    setCacheBlocks(false) stops the scan from evicting hot blocks.
  4. 4. In a Hive HBase mapping string, what does :key mean?

    :key maps a Hive column to the row key; other entries are family:qualifier.

Summary

  • Batch engines read one split per region; region count sets parallelism.
  • MapReduce uses TableMapper and TableReducer; Spark uses connectors or the input format; Hive uses an external table with a column mapping.
  • Scan jobs load the region servers; snapshot reads do not.
  • Always disable block caching for big scans.

Next: Thrift, REST and Phoenix