Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Filters and Coprocessors

Filter Lists and Custom Filters

Combine HBase filters with FilterList (MUST_PASS_ALL, MUST_PASS_ONE), understand ReturnCode ordering and deploy a custom filter, using Java stand-ins you can run.

One filter is rarely enough. A real query says "rows for this customer, with status error, that took longer than 500 ms, and only return two columns". HBase combines filters with a FilterList. When no built-in filter does what you need, you can write a custom filter class and ship it to the region servers.

This lesson explains how lists combine decisions, the return codes that make filters work, and what writing and deploying a custom filter involves. If you have not yet seen the basic filters, read Filters and Scan Basics first.

HBase cannot run in the live editors. Real filter classes are shown as read-only code. The runnable programs are Java stand-ins that mimic the decision logic (the filter return codes and list operators) so you can experiment with the rules.

FilterList: AND and OR

A FilterList has an operator and a list of filters:

  • MUST_PASS_ALL: a cell or row is kept only if every filter accepts it (AND).
  • MUST_PASS_ONE: it is kept if at least one filter accepts it (OR).

Lists nest, so you can express A AND (B OR C).

Java
// read-only: real HBase 2.x client
FilterList errorsOnly = new FilterList(FilterList.Operator.MUST_PASS_ALL);
errorsOnly.addFilter(new PrefixFilter(Bytes.toBytes("user42|")));

SingleColumnValueFilter err = new SingleColumnValueFilter(
    Bytes.toBytes("d"), Bytes.toBytes("status"), CompareOperator.EQUAL, Bytes.toBytes("error"));
err.setFilterIfMissing(true);

SingleColumnValueFilter slow = new SingleColumnValueFilter(
    Bytes.toBytes("d"), Bytes.toBytes("ms"), CompareOperator.GREATER, Bytes.toBytes(500L));
slow.setFilterIfMissing(true);

FilterList either = new FilterList(FilterList.Operator.MUST_PASS_ONE, err, slow);
errorsOnly.addFilter(either);
scan.setFilter(errorsOnly);

Order matters for speed. In MUST_PASS_ALL, HBase stops evaluating as soon as one filter rejects, so put cheap and selective filters (a prefix check) first and expensive ones (regex on values) last.

ReturnCode: the filter protocol

A filter is evaluated at three levels while the region server reads cells in order. The method that matters most is filterCell(Cell) (called filterKeyValue in older versions), which returns a ReturnCode:

ReturnCode Meaning
INCLUDE Keep this cell
SKIP Drop this cell, go to the next cell
NEXT_COL Skip the rest of this column's versions
NEXT_ROW Skip the rest of this row
SEEK_NEXT_USING_HINT Jump ahead to a cell returned by getNextCellHint
INCLUDE_AND_NEXT_COL Keep this cell, then skip remaining versions of the column

Return codes are optimizations as well as decisions: returning NEXT_ROW when a row can no longer match saves reading the row's remaining cells, and SEEK_NEXT_USING_HINT lets a filter skip large key ranges, which is how MultiRowRangeFilter and FuzzyRowFilter run fast.

There are also row-level hooks: filterRowKey (decide from the key alone, cheapest), filterRow() (final yes/no once all cells of a row were seen) and filterAllRemaining() (stop the whole scan, as PageFilter does when the page is full).

A stand-in for the protocol

The program models a mini scan engine with a Filter interface, a few filters and a list implementation. It counts how many cells each ordering of filters inspects.

Java FilterList and return codes (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

When the cheap prefix filter runs first and returns NEXT_ROW for row a|1, the engine skips its remaining three cells, so the expensive filter is called 8 times instead of 10 (the run prints both counts). The gap is small with 12 cells and grows with every row the cheap filter can reject early, which is the practical reason to put selective filters first and to return NEXT_ROW as early as possible.

Writing a custom filter

If you need logic that no built-in filter has, extend FilterBase. A custom filter has three pieces:

  1. The filter class, overriding filterCell (or filterRowKey, filterRow).
  2. Serialization: toByteArray() and a static parseFrom(byte[]), because the filter object is sent from the client to every region server. In HBase 2.x this uses Protocol Buffers.
  3. Deployment: the JAR must be on each region server's classpath, usually in hbase.dynamic.jars.dir or the server lib folder, followed by a rolling restart or dynamic load.
Java
// read-only sketch
public class LongerThanFilter extends FilterBase {
    private final long limit;
    public LongerThanFilter(long limit) { this.limit = limit; }

    @Override
    public ReturnCode filterCell(Cell c) {
        long v = Bytes.toLong(c.getValueArray(), c.getValueOffset());
        return v > limit ? ReturnCode.INCLUDE : ReturnCode.NEXT_ROW;
    }
    @Override public byte[] toByteArray() { return Bytes.toBytes(limit); }
    public static LongerThanFilter parseFrom(byte[] bytes) { return new LongerThanFilter(Bytes.toLong(bytes)); }
}

Custom filters are risky: a bug runs inside the region server, a broken class on the classpath can make scans fail, and each upgrade of HBase may change the filter API. Many teams avoid them, doing the work in the client, in a Spark job, or in a coprocessor instead.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does FilterList.Operator.MUST_PASS_ONE mean?

    MUST_PASS_ONE is an OR of the filters; MUST_PASS_ALL is an AND.
  2. 2. Which return code skips the remaining cells of the current row?

    NEXT_ROW tells the scanner to jump to the next row without evaluating the rest.
  3. 3. Why put cheap, selective filters first in a MUST_PASS_ALL list?

    AND evaluation short-circuits on the first rejection, so the order affects cost but not the result.
  4. 4. What is needed to use a custom filter?

    The filter is serialized and sent to the region servers, which must have the class on their classpath.

Summary

  • FilterList combines filters with AND (MUST_PASS_ALL) or OR (MUST_PASS_ONE) and can nest.
  • Return codes both decide and optimize: NEXT_ROW, NEXT_COL and seek hints skip work.
  • Put cheap, selective filters first.
  • Custom filters need serialization and server-side deployment, so use them sparingly.

Next: Coprocessors: Observers