Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: A Time-Series Metrics Store

Build a metrics store on an HBase schema: hourly row buckets, rollup tables with TTL, tag encoding, range queries and downsampling, tested on a Java stand-in.

Monitoring systems such as OpenTSDB were built on HBase because time series match its strengths: huge write volume, append-only data, range scans by metric and time. In this project you design and test the core of a metrics store: ingest points, query a time range, and keep long history cheaply with rollups and TTL.

HBase cannot run in the live editors. The runnable program uses a plain Java sorted map as a stand-in table; the same keys and operations are what you would run against HBase with the client API, shown read-only where useful.

Requirements

  1. Ingest points (metric, tags, timestamp, value) at high rates, e.g. 1 point per 10 s from each of 50,000 hosts.
  2. Query one metric for a host over a time range, returned in order.
  3. Keep raw data for 7 days, 1-minute averages for 90 days, 1-hour averages for 2 years.
  4. Writes must not hotspot.

Row key and layout

From Schema Patterns: one row per series per hour, one column per point offset.

TEXT
key    = salt(1 byte) | metricId (3 bytes) | hourStart (4 bytes) | tagHash (4 bytes)
family = d
qual   = offset within the hour in seconds (2 bytes)
value  = double (8 bytes)
  • Salt is hash(metricId + tags) % N, so each series sticks to one bucket but series spread across regions. (OpenTSDB uses a similar scheme.)
  • metricId is a short id from a lookup table: metric names like cpu.load.1m are long and are repeated in every cell otherwise (key length).
  • hourStart as a fixed-width big-endian integer sorts correctly.
  • Tags (host=web1) are replaced by ids too; here a hash keeps the example small.

Rollups use separate tables (metrics_raw, metrics_1m, metrics_1h) so each has its own TTL:

TEXT
create 'metrics_raw', {NAME => 'd', TTL => 604800,   COMPRESSION => 'ZSTD', DATA_BLOCK_ENCODING => 'FAST_DIFF'}, {SPLITS => ...}
create 'metrics_1m',  {NAME => 'd', TTL => 7776000,  COMPRESSION => 'ZSTD'}
create 'metrics_1h',  {NAME => 'd', TTL => 63072000, COMPRESSION => 'ZSTD'}

TTL on each table drops old data at compaction time (column families and versions) with no delete job.

Ingest, query and rollup

Java Metrics store with rollups (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The 720 raw points become 120 one-minute cells and 3 or fewer hourly cells. The hourly bucket boundaries in this model fall on ts - ts % 3600, so a two-hour window that does not start on the hour touches three buckets. Real systems rollup from the next-finer table rather than re-reading raw data, as the example does for the hour table.

Run rollups as batch jobs

Rollups are computed by a periodic job (Spark or MapReduce over HBase snapshots, see HBase with MapReduce, Spark and Hive) or by streaming aggregation before the data is written. Doing it in an observer or endpoint is possible but ties heavy computation to the region servers.

Averages roll up correctly only when you store sums and counts, not just averages (endpoint aggregation lesson). A better rollup cell holds sum, count, min and max in separate qualifiers.

Operational notes

  • Late data: a point arriving after its hour was rolled up must trigger a re-rollup of that bucket. Keep a "dirty hour" queue.
  • Cardinality: the number of distinct series (metric times tag sets) is the real scalability limit, not the number of points. Cap tags with unbounded values (request ids).
  • Compaction: TTL expiry makes old HFiles drop out at compaction; the date-tiered compaction policy fits this workload because old files stop being rewritten.
  • Scans: for dashboards scanning many series, use setCacheBlocks(true) only for hot recent hours.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why put one hour of points in one row?

    Bucketing reduces key repetition and rows, and range queries read contiguous hours.
  2. 2. Why use separate tables for raw and rolled-up data?

    TTL and storage settings are per column family; separate tables give different retention per resolution.
  3. 3. What should a rollup cell store so averages can be re-aggregated correctly?

    Averages cannot be merged; sums and counts can.
  4. 4. What is usually the real scalability limit of a metrics store?

    Every unique metric and tag combination is a series; unbounded tag values explode the series count.

Summary

  • Hourly row buckets with offset qualifiers reduce rows and speed time-range reads.
  • Salt by series, short ids for metric and tags, fixed-width big-endian times.
  • Separate tables with TTLs for raw, 1-minute and 1-hour data; roll up with batch jobs storing sums and counts.
  • Watch cardinality and late data.

Next: Practice and Interview Questions