Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 43 ยท Searching

Sorting and Pagination

Sort Elasticsearch results, page with from and size, and walk deep result sets with search_after and a point in time. Learn why deep paging is costly.

A search can match millions of documents, but a page shows ten. Two questions follow: in what order do the hits come, and how does the user get to the next page? Elasticsearch answers the first with sort and the second with from/size for shallow pages and search_after for deep ones. The second answer matters more than it looks, because the obvious approach (just raise from) gets slower the deeper you go, and Elasticsearch stops you at 10,000 hits by default.

This lesson explains sorting rules, the cost of deep pagination, how search_after works with a point in time (PIT), and why scroll is now mostly history. Real requests are read-only. A JavaScript stand-in implements a cursor walk so you can see how search_after finds the next page without counting from the start.

Three ways to page

from + size Simple. Fine for the first pages.
search_after Cursor based. Fast at any depth.
scroll Old bulk export tool. Mostly replaced.

Use PIT with search_after for consistent deep paging.

1 / 2

Sorting

By default hits are sorted by _score, highest first. Add sort to use field values instead:

JSON
GET /products/_search
{
  "query": { "match": { "name": "headphones" } },
  "sort": [
    { "price": "asc" },
    { "_score": "desc" },
    { "id": "asc" }
  ]
}

Sort keys apply in order: price first, then score for equal prices, then id as the final tie-breaker. A few rules to remember:

  • You can sort on numbers, dates, booleans and keyword fields. You cannot sort on an analyzed text field, because it has many tokens per document (see Mappings and Field Types). The usual fix is a keyword multi-field such as name.raw.
  • Sorting uses doc values (a column-oriented structure), not the inverted index (see The Inverted Index). That is why it is fast, and why a field with doc_values: false cannot be sorted.
  • If you sort by a field and not _score, scores are not computed unless you add "track_scores": true. That saves time.
  • Missing values go last by default. Control that with "missing": "_first" or "_last" on the sort entry.
  • Arrays sort by their min (ascending) or max (descending) value; you can choose with "mode".
JSON
{
  "sort": [
    { "rating": { "order": "desc", "missing": "_last" } },
    { "created": "desc" }
  ]
}

from and size

size is how many hits to return (default 10) and from is how many to skip:

JSON
GET /products/_search
{
  "from": 20,
  "size": 10,
  "query": { "match_all": {} },
  "sort": [{ "price": "asc" }, { "id": "asc" }]
}

That is page 3 of a 10-per-page list. It is easy to build a pager on it, and it is fine for the first few pages.

The cost of going deep

Elasticsearch splits an index into shards (see Shards, Replicas and Clusters). To return hits 10,000 to 10,010 in a global order, each shard must produce its own top 10,010 hits. The coordinating node merges all those lists and throws away everything except the last ten. With 5 shards, that is 50,050 entries to sort for ten results, and the work grows linearly with from.

For that reason index.max_result_window is set to 10,000. A request where from + size is greater than that fails with an error. You can raise the limit, but you are only raising the memory and CPU bill. Real users almost never go to page 1,000 of a search result list; bots and export jobs do. Treat deep paging as a different problem from showing a page.

The sorting must also be deterministic. If two documents tie on the sort value, their relative order may differ between requests or between replicas, so a document can appear on two pages or on none. Always add a unique tie-breaker, such as a unique id field, as the last sort key.

search_after

search_after replaces "skip N" with "start after this one". You send the sort values of the last hit you saw, and each shard continues from there:

JSON
GET /products/_search
{
  "size": 10,
  "query": { "match_all": {} },
  "sort": [{ "price": "asc" }, { "id": "asc" }],
  "search_after": [49.9, "p-0350"]
}

Each shard only needs to return its next 10 hits after that point, no matter how deep you are. The cost per page stays flat. The values in search_after come from the sort array of the last hit in the previous response:

JSON
{
  "hits": {
    "hits": [
      {
        "_id": "โ€ฆ",
        "_source": { "name": "โ€ฆ" },
        "sort": [49.9, "p-0350"]
      }
    ]
  }
}

The limits: you can only move forward (or backward by reversing every sort order), you cannot jump to page 57, and the sort needs a unique tie-breaker so the cursor points to exactly one position.

Point in time

Between two requests, documents may be indexed, updated or deleted, and refreshes change which segments are visible (see Segments, Refresh and Near-Real-Time Search). That can make a cursor skip or repeat documents. A point in time (PIT) freezes the view of the data:

JSON
POST /products/_pit?keep_alive=1m

The response contains an id. You pass it in the search instead of an index name:

JSON
GET /_search
{
  "size": 10,
  "pit": { "id": "46ToAwMDaWR5BXV1aWQyโ€ฆ", "keep_alive": "1m" },
  "sort": [{ "price": "asc" }, { "id": "asc" }],
  "search_after": [49.9, "p-0350"]
}

Every response may return an updated PIT id; use the latest one. When you finish, close it:

JSON
DELETE /_pit
{ "id": "46ToAwMDaWR5BXV1aWQyโ€ฆ" }

A PIT keeps old segments alive, so do not leave many open for long. The keep_alive only needs to cover the time between two requests, not the whole walk. If you add "track_total_hits": false to deep pages you also skip counting the total, which is another saving. Elasticsearch adds a hidden tiebreaker (_shard_doc) when you search with a PIT and do not provide one, but an explicit unique field makes the order meaningful and portable.

Try it: a search_after cursor walk

The stand-in has 5,000 product records on 3 pretend shards. It pages with from/size and with search_after, counting how many entries each shard has to produce. The second number is the "work" that makes deep paging expensive.

JS from/size versus search_after

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Both approaches return the same page, but from/size makes every shard collect 400 entries (1,200 in all) for ten results, and the cost keeps growing with the page number. The cursor version collects ten entries per shard (30 in all) every time.

Scroll, and why it is mostly history

Before PIT existed, scroll was the way to read a big result set. You started a scroll with ?scroll=1m, got a _scroll_id, and kept asking for the next batch with that id. It keeps a frozen search context open, much like a PIT. It still exists and is used by tools such as reindex, but for new code the documentation recommends PIT with search_after: it is more flexible (you can re-sort and run searches in parallel slices) and its cursor values are visible in each response.

Use scroll-like bulk reads only for export and reprocessing jobs, never to serve user pages. A human-facing page should:

  • use from/size for the first pages (a limit of a few hundred hits is typical),
  • offer "next/previous" using search_after when users really need to go deeper,
  • encourage narrowing the search with filters (see Faceted Search and Filters) instead of paging forever.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does from: 9990, size: 10 cost much more than from: 0, size: 10?

    To find a global position, every shard returns from + size entries. The work grows with from.
  2. 2. What is the default value of index.max_result_window?

    Requests where from + size is above 10,000 fail unless you raise the setting, which only moves the cost to memory and CPU.
  3. 3. Why should the sort for search_after end with a unique field such as an id?

    If several documents share the same sort values, the cursor cannot tell which of them you have already seen.
  4. 4. What does a point in time (PIT) add to search_after paging?

    A PIT keeps the segments that existed when it was opened, so a long walk sees a consistent snapshot.
  5. 5. Can you sort on an analyzed text field?

    A text field has many tokens per document, and sorting needs doc values. A keyword sub-field gives one value per document.

Summary

  • sort orders hits by doc values; sort keys apply in order and you should end with a unique tie-breaker.
  • from/size is simple but each shard collects from + size hits, so cost grows with depth, and max_result_window caps it at 10,000.
  • search_after takes the sort values of the last hit as a cursor, so each page costs about the same.
  • A PIT freezes the data during a walk; close it when finished.
  • scroll still works for exports but PIT plus search_after is the modern choice.

Next: Highlighting, Autocomplete and "Did You Mean".