Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Administration and Tuning

Compactions and Flushes

How HBase turns writes into HFiles: MemStore flushes, minor and major compactions, write amplification and the settings to tune, with a runnable LSM simulation.

HBase is a log-structured merge (LSM) store. Writes never modify files in place. They go to the write-ahead log (WAL) and to an in-memory buffer, the MemStore; when the MemStore is full it is flushed to a new immutable file on disk, an HFile. Over time many small files pile up, so background compactions merge them into fewer, larger files. Understanding this cycle explains most of the performance behavior you will see in production: write stalls, read slowdowns, disk spikes and why deletes do not free space at once.

HBase cannot run in the live editors on this site. Configuration and shell commands are shown read-only. The runnable Java blocks are labelled simulations of the flush and compaction logic, not HBase code.

The write path in one picture

flowchart LR
  P["Put"] --> W["WAL (sync to HDFS)"]
  P --> M["MemStore (sorted, in memory)"]
  M -->|"full: flush"| H1["HFile 1"]
  M -->|"full: flush"| H2["HFile 2"]
  H1 --> C["Compaction"]
  H2 --> C
  C --> H3["HFile 3 (merged)"]

Each column family of each region has its own MemStore and its own set of HFiles.

The WAL protects against a crash before the flush: after a region server failure, the log is replayed to rebuild lost MemStores. The WAL is not read during normal operation.

Flushes

A flush writes the sorted contents of a MemStore as a new HFile. Triggers:

  • The MemStore reaches hbase.hregion.memstore.flush.size (default 128 MB per region per family).
  • Total MemStore memory on a server reaches the global limit (hbase.regionserver.global.memstore.size, default 40% of the heap); the largest MemStores flush first.
  • Too many WAL files exist (the oldest data must be flushed so the log can be archived).
  • You request it: flush 'table' in the shell.

While a MemStore flushes, writes continue into a fresh one. If flushes cannot keep up and a MemStore grows past a multiple (hbase.hregion.memstore.block.multiplier, default 4) of the flush size, the region blocks updates until the flush finishes. That is the "write stall" you see as latency spikes.

Why files must be merged

A read must check the MemStore and every HFile that might contain the row (Bloom filters help skip files, see MemStore, Block Cache and Bloom Filters). The more HFiles a store has, the slower reads become. Compaction reduces the file count and also applies deletes and drops expired or extra versions.

Minor and major compactions

  • A minor compaction picks a few small neighboring HFiles and rewrites them as one. It does not remove deleted data and expired cells in general, because it cannot be sure older files do not hold matching versions.
  • A major compaction rewrites all HFiles of a store into one. Only then are tombstones, expired cells (TTL) and surplus versions dropped for good.

Major compactions are expensive: they read and write the entire store. By default they run every 7 days (hbase.hregion.majorcompaction, with jitter). Many production clusters disable the automatic one (0) and trigger it during quiet hours, per table:

TEXT
major_compact 'events'
compact 'events', 'd'          # a minor compaction request for family d

Compaction selection in modern HBase uses the ratio-based policy (ExploringCompactionPolicy): a set of files is chosen only if the file sizes are balanced enough, controlled by hbase.hstore.compaction.ratio (default 1.2), hbase.hstore.compaction.min (3) and max (10) files. Date-tiered and stripe compaction policies exist for time-series and very large regions.

Write amplification

Each cell is written once to the WAL, once at flush and then again every time a compaction rewrites the file it lives in. If every byte is rewritten five times, your disks write five times more than the clients sent. A major compaction can double disk usage temporarily, because the old files remain until the new one is complete.

Simulating the cycle

The simulation below inserts keys into a MemStore with a tiny limit, flushes to HFiles (sorted lists), compacts when there are four files, and counts the cells written. It also shows a delete surviving a minor compaction and disappearing in a major one.

Java MemStore, flush and compaction (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The numbers show the point: the client wrote ten cells, but the disk wrote more because every compaction rewrites the data. The tombstone for row2 is kept by the minor compaction and removed by the major one.

Tuning guidelines

  • Do not shrink flush sizes to "save memory": more, smaller files mean more compaction work.
  • Watch the store file count per region and the compaction queue size in the metrics. A growing queue means compactions cannot keep up (more disks, slower writes, or a better key design).
  • Avoid running major compactions on a busy cluster; schedule them and use hbase.offpeak.start.hour and end.hour for off-peak throttling.
  • Throttle compaction throughput on shared clusters (hbase.regionserver.throughput.controller).
  • After deleting a lot of data, a major compaction is what actually frees disk.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is flushed when a MemStore fills up?

    A flush writes the MemStore to a new HFile; older files are never edited.
  2. 2. Which compaction permanently removes tombstones and expired cells?

    Only a major compaction sees all files of a store, so it can safely drop deleted and expired data.
  3. 3. Why do too many HFiles slow reads?

    More files mean more places a cell may live; compaction reduces how many must be checked.
  4. 4. What causes write stalls on a region?

    When flushing cannot keep up, the region blocks updates until memory is released.

Summary

  • Writes go to the WAL and MemStore; flushes create immutable HFiles.
  • Compactions merge files; major compactions are the only ones that purge deletes, TTL and extra versions.
  • Compaction rewrites data repeatedly, so disk throughput matters.
  • Watch store file counts and the compaction queue; schedule major compactions off-peak.

Next: Region Splitting and Balancing