Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Schema Design

Column Families and Versions

Design HBase column families, qualifiers, cell versions and TTL: how many families to use, what versions cost, and how timestamps drive reads.

An HBase cell is addressed by four coordinates: row key, column family, column qualifier and timestamp. The row key decided where the data lives (Row Key Design). This lesson is about the other three: how to organize columns into families, how to name qualifiers, and how to use versions (timestamps) and TTL on purpose instead of by accident.

Real HBase is not available in the live editors. The runnable blocks are Java stand-ins that model a versioned cell store with a sorted map. The real shell and client code is shown read-only.

Column families are physical

A column family is not just a label. Each family of each region gets its own store: its own MemStore, its own set of HFiles on disk, its own compactions, compression, block size, Bloom filter type, TTL and maximum versions. Families are declared up front in the table schema and are expensive to change; qualifiers inside a family are created freely at write time.

TEXT
Table: users
  Family "p"  (profile)   qualifiers: name, email, city        -> small, read often
  Family "a"  (activity)  qualifiers: last_login, page_views   -> written constantly
  Family "x"  (avatar)    qualifiers: image                    -> large values, rarely read

Splitting by access pattern lets you read p without touching the big x files, and compress or TTL each family differently.

Rules of thumb for families

  • Keep it to one to three families. Each family adds a MemStore and files per region. When one family flushes, its neighbors flush too in older versions of HBase, producing many small files if the families grow at very different rates.
  • Use short names. The family name is stored in every cell. p is better than profile_information.
  • Put together what is read together. If two groups of columns are almost always queried in the same Get, a single family is simpler and faster.
  • Separate by lifetime and size. Logs with a 7-day TTL and permanent profile data belong in different families.

Qualifiers are data

Qualifiers are arbitrary bytes created on write. Rows may have completely different qualifiers, and empty cells cost nothing, which is what "sparse" means in HBase. This makes some designs possible that a fixed-column database cannot do, such as using the qualifier to hold a value:

TEXT
row "user42", family "f" (followers)
  f:user7   = 1
  f:user19  = 1
  f:user88  = 1

Here the qualifier is the follower's id and the value is a marker. A scan with a column prefix filter lists followers. Like row keys, qualifiers are repeated in every cell, so keep them short.

Versions and timestamps

Every cell carries a timestamp, a long that defaults to the write time in milliseconds. A column family keeps up to VERSIONS values per cell (default 1 in HBase 2.x), newest first. A Get returns the newest version unless you ask for more or for a time range.

Java A versioned cell store (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The old write at timestamp 50 is stored and then trimmed straight away, because only three newer versions exist. That is also how real HBase behaves in principle: the timestamp, not arrival order, decides what is newest. (In real HBase the trimming happens during flushes and compactions, and a Get with readVersions(n) can still see extra versions until then.)

Real code to read versions, read-only:

Java
Get get = new Get(Bytes.toBytes("user42"))
    .addColumn(Bytes.toBytes("p"), Bytes.toBytes("city"))
    .readVersions(3)                                   // up to 3 versions
    .setTimeRange(200L, 400L);                         // [200, 400)
Result r = table.get(get);
for (Cell c : r.getColumnCells(Bytes.toBytes("p"), Bytes.toBytes("city"))) {
    System.out.println(c.getTimestamp() + " " + Bytes.toString(CellUtil.cloneValue(c)));
}

Choosing a version count

VERSIONS => 1 is right for most columns that hold the current state. Raise it only when you actually read history. Because versions live in the same HFiles, a high count multiplies storage and slows scans. Avoid using versions as an unbounded log (millions of versions of one cell); model that as separate rows instead, with a time component in the key.

Be careful when you set your own timestamps. Mixing client-set and server-set timestamps in one column leads to confusing results, because a cell written with a far-future timestamp hides later normal writes until compaction.

TTL and deletes

TTL is set per column family in seconds. Cells older than the TTL are invisible to reads and physically removed by compactions. It is the cheapest way to age out logs and sessions.

TEXT
alter 'events', {NAME => 'd', TTL => 604800}       # 7 days
create 'users', {NAME => 'p', VERSIONS => 1}, {NAME => 'a', VERSIONS => 1, TTL => 2592000}

A delete in HBase does not remove data immediately: it writes a tombstone marker that hides older cells, and the data is removed at the next major compaction. Three levels exist: delete a single version, delete all versions of a column, and delete a whole family or row. This is why a table can grow after you delete many rows, until compaction runs (see Compactions and Flushes).

Java TTL and tombstones (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The delete marker plus the expired cells hide everything for readers at once, but disk space is only reclaimed by the compaction step.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which settings are configured per column family?

    Each family has its own store, so storage properties such as TTL, compression, Bloom filter type and VERSIONS are family-level.
  2. 2. Why prefer one to three families per table?

    Each family has its own MemStore and HFiles in every region, so many families add overhead and flush pressure.
  3. 3. A cell has versions at timestamps 100, 200, 300 and VERSIONS is 2. What does a plain Get return?

    Get returns the newest version by timestamp, which is 300.
  4. 4. What does deleting a row do immediately on disk?

    HBase writes delete markers (tombstones); the data is physically removed at major compaction.

Summary

  • Families are physical stores: separate access patterns and lifetimes, keep them few and short-named.
  • Qualifiers are created on write, can carry data, and are repeated in each cell.
  • The timestamp decides which version is newest; set VERSIONS to what you read.
  • TTL ages data out; deletes are tombstones removed at major compaction.

Next: Schema Patterns: Time Series and Wide Tables