Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Architecture and Data Model

The Write Path: WAL, MemStore and HFiles

Follow a single HBase write from the client to disk: the WAL, the MemStore, flushes into HFiles, minor and major compactions, and why writes stay fast.

HBase is famous for fast writes, even on ordinary disks. The reason is that a write almost never touches a sorted file on disk. Instead it is appended to a log and put in memory. Sorting, merging and rewriting happen later in the background. This design is called a log-structured merge tree (LSM tree), and understanding it explains compaction, flushes, write stalls and most tuning knobs.

Following one put

sequenceDiagram
  participant C as Client
  participant R as RegionServer
  participant W as WAL (HDFS)
  participant M as MemStore
  C->>R: Put(row, family:qualifier, value)
  R->>W: append edit
  W-->>R: synced to replicas
  R->>M: insert into sorted memory
  R-->>C: success
  Note over M: later: flush to a new HFile
  1. The client finds the right RegionServer (see The Master and RegionServers) and sends the put.
  2. The RegionServer takes a row-level lock, assigns a timestamp if none was given and a sequence id.
  3. The edit is appended to the WAL and synced so it survives a crash. The WAL is a sequential write, the cheapest kind of disk I/O.
  4. The edit is inserted into the MemStore of the target column family, a sorted in-memory structure.
  5. The server replies to the client. No HFile has been written yet.

If the server crashes now, the MemStore is lost, but the WAL is not. The Master hands the region to another server, which replays the WAL edits into a new MemStore. This is why the WAL gives durability.

You can trade durability for speed per mutation. Durability.SKIP_WAL skips the log, which is fast but loses data on a crash; ASYNC_WAL syncs shortly after; SYNC_WAL (the default) syncs before replying; FSYNC_WAL forces a disk sync. Choose SKIP_WAL only for data you can recompute.

Flushing the MemStore

The MemStore cannot grow forever. When one reaches hbase.hregion.memstore.flush.size (128 MB by default), or when the sum of all memstores on a server crosses a share of the heap, a flush writes it out as a new sorted, immutable HFile in HDFS. Edits in the WAL that belong to flushed data are no longer needed, so old WAL files can be archived.

An HFile is a sorted sequence of data blocks (64 KB by default) with an index and optional bloom filters at the end. It is never modified after creation.

If writes arrive faster than flushing can keep up, HBase protects itself by blocking writes to the region (RegionTooBusyException or a delay) until the flush completes. Watching for blocked updates in the metrics tells you your memstore settings or disks are undersized.

The pile of files and compaction

After many flushes, a Store has many small HFiles. A read may have to look in all of them, so reads would get slower and slower. Compaction fixes that by merging files.

  • Minor compaction picks a few small, neighboring HFiles and merges them into one larger file. It reduces the file count but does not remove deleted data completely.
  • Major compaction rewrites all the HFiles of a Store into one. It physically drops tombstones, expired cells (TTL) and versions beyond the limit. It is heavy on I/O. By default it runs about every 7 days (with random jitter), and many production teams disable the automatic schedule and run it in quiet hours.

Compaction rewrites the same data several times over its life. That is called write amplification: the price you pay for fast writes and fast sorted reads.

Java Stand-in: a tiny LSM tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Crash recovery in miniature

The next stand-in shows why the WAL matters: data in the MemStore is lost on a crash, and the WAL rebuilds it. Remove the WAL replay line and see what would be missing.

Java Stand-in: recovering a MemStore from the WAL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Bulk loading skips the path

Normal writes pay for the WAL and the MemStore. For one-off imports of very large data sets you can generate HFiles directly (for example with a MapReduce or Spark job using HFileOutputFormat2) and then "load" them into the table with LoadIncrementalHFiles / BulkLoadHFiles. The files are simply moved into the region directories: no WAL, no memstore, almost no load on the RegionServers. The cost is that the data appears all at once and is not covered by the normal durability mechanisms until it is in place.

Settings worth knowing

  • hbase.hregion.memstore.flush.size: per-memstore flush trigger (default 128 MB).
  • hbase.regionserver.global.memstore.size: share of heap all memstores may use (default 0.4).
  • hbase.hstore.compactionThreshold: how many HFiles trigger a minor compaction (default 3).
  • hbase.hstore.blockingStoreFiles: when a Store has this many files, writes block until compaction catches up (default 16).
  • hbase.hregion.majorcompaction: major compaction interval in milliseconds (default 7 days).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In what order does a normal put proceed?

    The edit is made durable in the WAL first, then placed in the sorted MemStore. HFiles are written later by a flush.
  2. 2. What does a flush do?

    Flushes turn in-memory data into sorted HFiles in HDFS.
  3. 3. Which operation physically removes tombstones and expired cells?

    Only a major compaction rewrites all files of a Store, so it can safely drop deletes, expired cells and surplus versions.
  4. 4. What is the risk of Durability.SKIP_WAL?

    Without the WAL there is nothing to replay after a crash.
  5. 5. Why does compaction cause write amplification?

    Merging HFiles rewrites already-written data; that extra I/O is the cost of the LSM design.

Summary

  • A put goes to the WAL (durability), then to the sorted MemStore, then the client gets its answer.
  • Flushes write MemStores as immutable HFiles; the WAL entries for flushed data can be discarded.
  • Minor compactions merge a few files, major compactions rewrite everything and drop deleted data.
  • The LSM design trades extra background I/O (write amplification) for fast writes and sorted reads.
  • Durability, memstore size and compaction settings are the main write-path tuning knobs.

Next: The Read Path: BlockCache and Bloom Filters