Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 36 ยท Filters and Coprocessors

Endpoint Coprocessors and Aggregation

Learn how HBase endpoint coprocessors run per-region computations and how clients combine the partial results, including AggregationClient, with a runnable Java model.

An observer reacts to events. An endpoint coprocessor is different: it adds a new remote method to a table. The client calls the method on one region or on every region in a key range, each region computes a partial answer next to its data, and the client combines the partial answers. It is HBase's version of a stored procedure, and the same idea as map and reduce.

The classic use is aggregation: count, sum, min, max, avg. Without an endpoint the client must scan every row over the network to compute a sum; with one, only a handful of numbers cross the network. Observers are covered in Coprocessors: Observers.

HBase is not available in the live editors. The real endpoint and client code is shown read-only. The runnable Java stand-in splits a table into regions and runs a "per-region compute, client merge" computation so you can see the data flow.

How an endpoint works

  1. You define a service in a Protocol Buffers .proto file: request and response messages and the methods.
  2. You implement the service in a class that is both a RegionCoprocessor and a protobuf Service. It runs inside each region and can read that region's rows directly through RegionCoprocessorEnvironment.getRegion().
  3. The client calls table.coprocessorService(...) (or batchCoprocessorService) with a start and end row. HBase routes one call to each region that overlaps the range, collects the results, and returns them in a map from region name to response.
  4. The client merges the responses.
flowchart TB
  Client["Client: sum over rows A to Z"] --> R1["Region 1 (A-H)<br/>partial sum"]
  Client --> R2["Region 2 (I-Q)<br/>partial sum"]
  Client --> R3["Region 3 (R-Z)<br/>partial sum"]
  R1 --> M["Client merges 3 numbers"]
  R2 --> M
  R3 --> M

Work happens in parallel inside each region; only partial results travel back.

A sketch of a real endpoint (read-only)

The .proto file:

TEXT
message SumRequest  { required bytes family = 1; required bytes qualifier = 2; }
message SumResponse { required int64 sum = 1; required int64 rows = 2; }
service SumService  { rpc getSum(SumRequest) returns (SumResponse); }

The server side, implemented inside the region:

Java
public class SumEndpoint extends SumProtos.SumService implements RegionCoprocessor {
    private RegionCoprocessorEnvironment env;

    @Override public Iterable<Service> getServices() { return Collections.singleton(this); }
    @Override public void start(CoprocessorEnvironment e) { env = (RegionCoprocessorEnvironment) e; }

    @Override
    public void getSum(RpcController controller, SumProtos.SumRequest request,
                       RpcCallback<SumProtos.SumResponse> done) {
        long sum = 0, rows = 0;
        Scan scan = new Scan().addColumn(request.getFamily().toByteArray(),
                                         request.getQualifier().toByteArray());
        try (InternalScanner scanner = env.getRegion().getScanner(scan)) {
            List<Cell> cells = new ArrayList<>();
            boolean more;
            do {
                more = scanner.next(cells);
                for (Cell c : cells) { sum += Bytes.toLong(CellUtil.cloneValue(c)); rows++; }
                cells.clear();
            } while (more);
        } catch (IOException e) {
            ResponseConverter.setControllerException(controller, e);
        }
        done.run(SumProtos.SumResponse.newBuilder().setSum(sum).setRows(rows).build());
    }
}

The client side:

Java
Map<byte[], SumProtos.SumResponse> results = table.coprocessorService(
    SumProtos.SumService.class, startRow, stopRow,
    service -> {
        BlockingRpcCallback<SumProtos.SumResponse> callback = new BlockingRpcCallback<>();
        service.getSum(null, request, callback);
        return callback.get();
    });
long total = 0;
for (SumProtos.SumResponse r : results.values()) total += r.getSum();

The built-in AggregateImplementation

You rarely write sum yourself. HBase ships AggregateImplementation (the server side) and AggregationClient (the client side) with rowCount, sum, max, min, avg and std. Enable the endpoint on the table and use a column interpreter that tells it how to read values (usually LongColumnInterpreter):

Java
// read-only
AggregationClient ac = new AggregationClient(conf);
Scan scan = new Scan().addColumn(Bytes.toBytes("d"), Bytes.toBytes("ms"));
long total = ac.sum(TableName.valueOf("events"), new LongColumnInterpreter(), scan);
long rows  = ac.rowCount(TableName.valueOf("events"), new LongColumnInterpreter(), scan);

An average is not the average of the regions' averages: each region returns its sum and count, and the client divides the totals. This "return mergeable partial state" rule applies to every aggregation you write yourself.

Per-region compute, client merge

Java Endpoint-style aggregation (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The last line usually differs from the correct average, because regions hold different numbers of rows. Always merge sums and counts, not finished averages.

When to use an endpoint

Endpoints help when the computation reduces data a lot (counts, sums, top-N per region) and the network is the bottleneck. They are risky for the same reasons as observers: your code runs in the region server, a long computation holds a handler thread, and a wrong implementation can destabilize the server. Heavy analytics belong in Spark or MapReduce over HBase snapshots (Integration); simple counts can use FirstKeyOnlyFilter with a scan. HBase 2.x keeps endpoints but disables loading them by default in some security setups, so check your cluster policy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an endpoint coprocessor return to the client?

    Each region computes next to its data and returns a small response; the client combines them.
  2. 2. How should an average be computed across regions?

    Averages of averages are wrong when regions differ in size; sums and counts are mergeable.
  3. 3. Where is the service for an endpoint defined?

    Endpoints are protobuf services; the client calls their methods over RPC.
  4. 4. Which built-in class provides sum, min, max and average endpoints?

    AggregateImplementation runs on the server and AggregationClient merges the per-region results.

Summary

  • Endpoints are server-side methods called explicitly, one call per region in a range.
  • The pattern is compute per region, then merge on the client.
  • Return mergeable partial state (sum and count), never finished averages.
  • AggregationClient covers counts and basic statistics; heavier analytics belong in batch engines.
  • Endpoints share the risks of observers: run in the server process, keep them short.

Next: Compactions and Flushes