Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 43 ยท Aggregations and Analytics

Sub-Aggregations and Pipeline Aggregations

Nest aggregations, fetch top_hits per bucket, page through buckets with composite, and compute moving averages, derivatives and bucket_script results.

One aggregation is useful. A tree of them is where Elasticsearch starts to feel like an analytics engine. Put an aggregation inside a bucket aggregation and it runs once per bucket: "average price per country", "the three priciest orders per category". Add a second family, pipeline aggregations, and you can compute on the results of other aggregations: a moving average of daily sales, the change from one month to the next, revenue divided by order count.

This lesson builds on Aggregations: Metrics and Buckets. The real requests are read-only here, and each idea has a runnable Python stand-in that does the same calculation with plain loops.

Three ways to combine aggregations

Nesting A bucket holds more aggregations.
Composite Walk through all buckets, page by page.
Pipeline Compute on other aggregation results.

Nesting multiplies buckets, so keep an eye on the total count.

1 / 2

Sub-aggregations

Every bucket aggregation can contain an aggs block of its own. The inner aggregations are evaluated separately for each bucket of the outer one.

JSON
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "by_category": {
      "terms": { "field": "category" },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } },
        "by_country": {
          "terms": { "field": "country", "size": 3 },
          "aggs": { "revenue": { "sum": { "field": "price" } } }
        }
      }
    }
  }
}

The response mirrors the request: each by_category bucket has its key, doc_count, avg_price and a nested list of by_country buckets. Depth is cheap to write and expensive to run. Ten categories times ten countries times ten cities is already a thousand leaf buckets, and Elasticsearch caps the total (search.max_buckets) to protect itself. Keep inner size values small and ask yourself whether you really need the cross product.

top_hits: the best documents per bucket

top_hits is a metric aggregation that returns actual documents. Inside a terms aggregation it answers "show me the two most expensive orders in each category", which a plain search with size cannot do.

JSON
"aggs": {
  "by_category": {
    "terms": { "field": "category" },
    "aggs": {
      "priciest": {
        "top_hits": { "size": 2, "sort": [{ "price": "desc" }], "_source": ["id", "price"] }
      }
    }
  }
}

Use it sparingly: it fetches stored documents for every bucket, so a large outer size with a large top_hits is heavy. A small size (1 to 5) is the norm, as in "latest article per author".

Python Nested terms, metrics and top_hits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output is the same tree the real engine would build for those three nested aggregations, minus the bookkeeping fields (doc_count_error_upper_bound, took, and so on). Notice that the inner avg_price and revenue values are computed only over the documents of their own bucket.

composite: stepping through every bucket

A terms aggregation shows only the top buckets. What if you need all of them, say to export "total spend per customer" for a million customers? The composite aggregation returns buckets in a stable order and gives you an after_key to resume from, much like a cursor.

JSON
GET /orders/_search
{
  "size": 0,
  "aggs": {
    "customers": {
      "composite": {
        "size": 1000,
        "sources": [
          { "country":  { "terms": { "field": "country" } } },
          { "category": { "terms": { "field": "category" } } }
        ]
      },
      "aggs": { "spend": { "sum": { "field": "price" } } }
    }
  }
}

Each bucket key is a combination of all the sources (country and category here). The response includes "after_key": { "country": "FR", "category": "toys" }. You repeat the request with "after": <that object> inside composite, and stop when a page comes back with no buckets. The order is by the source values, never by doc_count, which is the price you pay for being able to page through everything.

Python Paging buckets with after_key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loop is exactly what a client library does: ask for a page, remember the last key, ask again. Because each page starts strictly after the previous key, no bucket is skipped or repeated, even if you stop and continue later.

Pipeline aggregations

A pipeline aggregation takes the output of other aggregations as its input. It does not look at documents. There are two flavors. A parent pipeline (such as derivative, moving_avg-style smoothing, cumulative_sum, bucket_script) is placed inside a bucket aggregation and adds a new value to each bucket. A sibling pipeline (such as avg_bucket, max_bucket, stats_bucket) sits next to a bucket aggregation and reduces all its buckets to one number. You point at the input with buckets_path, a small path language like "sales>revenue".

JSON
"aggs": {
  "per_day": {
    "date_histogram": { "field": "day", "calendar_interval": "day" },
    "aggs": {
      "revenue": { "sum": { "field": "price" } },
      "orders":  { "value_count": { "field": "id" } },
      "avg_order_value": {
        "bucket_script": {
          "buckets_path": { "rev": "revenue", "n": "orders" },
          "script": "params.rev / params.n"
        }
      },
      "change": { "derivative": { "buckets_path": "revenue" } },
      "running_total": { "cumulative_sum": { "buckets_path": "revenue" } }
    }
  },
  "best_day": { "max_bucket": { "buckets_path": "per_day>revenue" } }
}

For smoothing, the moving_fn aggregation applies a function over a sliding window of earlier bucket values: "window": 3, "script": "MovingFunctions.unweightedAvg(values)". The first buckets have fewer than three values to look at, so the average is over what exists (check the exact edge behavior of your version before depending on it).

Two cautions. A derivative needs the buckets to be adjacent, so use "min_doc_count": 0 on the date_histogram and think about gap_policy (skip or insert_zeros) for empty days. And scripts cost something per bucket, so keep them for pipelines over a modest number of buckets.

Python Pipelines over daily revenue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first row has no derivative, just like the real aggregation, which has no earlier bucket to compare with. The moving average starts with a short window, and the running total ends at the sum of all buckets. Compare the avg3 column with revenue: on March 3 the raw revenue drops by 45 but the smoothed value drops only from 127.5 to 115.0, and from March 4 on it climbs steadily while the raw numbers jump up and down. The per order column is a flat 30.00 on most days, which is what bucket_script computed by dividing revenue by order count.

Approximate results, honestly

Distributed analytics trades a little exactness for speed. Remember these sources of approximation: terms on multi-shard indexes (see doc_count_error_upper_bound), cardinality and percentiles (sketches), and nested terms where the inner top-N is chosen per outer bucket. Pipelines compute on those results, so they inherit any error. When you need exact numbers for money, use composite over the grouping fields, or compute them in a database, and keep Elasticsearch for exploration and dashboards.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an aggregation placed inside a terms aggregation's aggs block do?

    Sub-aggregations are evaluated per bucket, which is how you get values like the average price per country.
  2. 2. Which aggregation lets you page through every bucket with an after_key?

    composite returns buckets in a stable order of its sources and gives an after_key to continue from. terms is limited to its top buckets.
  3. 3. Where does a derivative aggregation get its input?

    Pipeline aggregations work on aggregation output, not documents. buckets_path names the metric to differentiate.
  4. 4. Why should the size of nested terms aggregations be kept small?

    Bucket counts multiply across levels (10 x 10 x 10 = 1,000), so deep nesting with large sizes becomes expensive quickly.

Summary

  • Sub-aggregations run once per bucket; keep nesting shallow and inner sizes small.
  • top_hits returns real documents per bucket, so use a small size.
  • composite produces ordered buckets and an after_key, which is the way to read all buckets.
  • Parent pipelines (derivative, cumulative_sum, bucket_script, moving_fn) add values to buckets; sibling pipelines (max_bucket, avg_bucket) summarize all buckets.
  • Use min_doc_count: 0 and a gap_policy so empty buckets do not distort pipelines, and remember that approximate inputs give approximate outputs.

Next: Faceted Search and Filters.