Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 43 ยท How Search Works

Segments, Refresh and Near-Real-Time Search

Learn why a new Elasticsearch document is not searchable instantly: in-memory buffers, refresh, the translog, flush, segment merges and lazy deletes.

You index a document, get a 201 Created back, search for it a moment later, and it is not there. Then, a second later, it is. This is not a bug. Elasticsearch is near-real-time: a document becomes searchable shortly after it is written, not at the exact moment. To understand why, you need to know how a shard stores data on disk and in memory.

In the previous lessons you saw that a Lucene index is built from immutable pieces called segments. This lesson explains how new documents become segments (refresh), how they stay safe in a crash (translog and flush), and how many small segments are combined into fewer big ones (merge).

The real engine cannot run on this site, so the REST calls below are read-only. The runnable examples are small JavaScript models of the same ideas. They show the logic, not the real performance or file formats.

The write path in one picture

A write goes to two places

Memory buffer Holds new documents, not yet searchable.
Translog Append-only log on disk for crash safety.

Both are written before the request returns.

1 / 4

When a document arrives at a shard, two things happen. It is added to an in-memory indexing buffer, and the operation is appended to the translog (transaction log), a file on disk. By default the translog is synced to disk before the request is acknowledged (index.translog.durability: request), so an acknowledged write survives a crash.

At this point the document is not searchable. The buffer is not a segment yet, and searches only look at segments.

Refresh: the step that makes documents visible

A refresh writes the buffer out as a new segment and opens it for searching. It is cheap compared with a full commit, because the new segment is first held in the operating system's file cache rather than being forced to disk.

The default index.refresh_interval is 1s, which is why "near-real-time" usually means about one second. One more detail: if a shard has not been searched for a while (30 seconds by default), Elasticsearch skips the scheduled refreshes until the next search arrives, and that search waits for a refresh. This saves work on idle indexes.

You can control visibility yourself:

HTTP
POST /products/_refresh

PUT /products/_doc/1?refresh=wait_for
{ "name": "Trail shoe" }

PUT /products/_settings
{ "index": { "refresh_interval": "30s" } }
  • POST /products/_refresh refreshes now.
  • ?refresh=wait_for makes the request return only after the next refresh, so the document is searchable when you get the response.
  • ?refresh=true forces an immediate refresh. It works, but doing it on every write creates a tiny segment per request and hurts performance.
  • A longer refresh_interval (or -1 to disable it during a big import) makes bulk loading faster.

One exception to the delay: a get by ID (GET /products/_doc/1) is real-time. It can read a not-yet-refreshed document from the translog, so it sees your write immediately. Searches do not.

JS A shard with a buffer, a translog and segments

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first search finds nothing, even though two documents were accepted. After the refresh both are found. Document c is invisible until the flush (which refreshes first) makes it a segment. The translog is emptied by the flush because everything it protected is now in a segment.

Flush: committing to disk

Segments created by a refresh may live only in the file cache. A flush performs a Lucene commit: it forces the segments to disk, records which segments belong to the shard, and starts a new translog generation. After a crash, Elasticsearch restarts from the last commit and replays the translog to recover the operations that came after it.

Elasticsearch flushes automatically, for example when the translog grows past index.translog.flush_threshold_size (512 MB by default). You rarely call POST /products/_flush yourself.

Do not confuse the two: refresh = searchable, flush = durable and translog cleared. Durability of an acknowledged write comes from the translog, not from the flush.

Segments are immutable

A segment is never edited after it is written. That makes it simple and fast: no locks while searching, easy caching, compact compression. The price is that you cannot change or remove a document inside a segment.

  • Delete: the document is only marked as deleted in a small per-segment record. It stays physically in the segment. Searches skip marked documents.
  • Update: Elasticsearch marks the old version as deleted and indexes a whole new document. An update is a delete plus an insert.

This is why deletes are called lazy: they free no disk space right away, and a heavy update workload creates many dead documents.

Merging: cleaning up in the background

Every refresh adds a segment, and each search must visit every segment. A background merge reads several segments, copies only their live documents into one new, larger segment, and deletes the old ones. Merging is what finally removes deleted documents from disk. The default policy (tiered merging) picks similarly sized segments and runs on its own, throttled so it does not starve searches.

A side effect: while deleted documents still sit in segments, they still count in some statistics, such as the document counts used by BM25 scoring. Scores can shift slightly after a merge.

JS Lazy deletes and merging

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that after the delete and the update the number of dead documents grows, and the number of segments grows too. Only the merge brings both down. This is the stand-in for what Lucene does in the background, and it explains why an index with constant updates can use much more disk than its live data suggests.

Force merge: a special tool

POST /products/_forcemerge?max_num_segments=1 merges a shard down to a chosen number of segments. It can speed up searches on an index that will never change again (for example last month's logs), but running it on an index that is still receiving writes is a bad idea: it creates a huge segment that normal merging then struggles to reshape.

Practical advice

  • Do not call refresh after every write in application code. Use refresh=wait_for only where a user must see the result at once (for example after saving a form), and prefer showing the saved data from your own response.
  • Raise refresh_interval (or set -1) during large imports and reset it afterward.
  • Expect a short delay between write and search. Design screens accordingly.
  • Avoid workloads that update the same documents very frequently; every update is a new document plus a dead one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a document not found by a search right after the index request returns?

    New documents sit in the indexing buffer and the translog. A refresh (every second by default) writes them as a segment that searches can see.
  2. 2. What does the translog protect against?

    After a crash, Elasticsearch replays the translog on top of the last commit, so operations that were acknowledged but not flushed are recovered.
  3. 3. What really happens when you delete a document?

    Segments are immutable. Deletes are lazy markers, and a merge copies only live documents into a new segment.
  4. 4. Which request is real-time, so it can see a document before the next refresh?

    Get by ID can read from the translog. Search requests only see refreshed segments.

Summary

  • Writes go to an in-memory buffer and the translog; they are searchable only after a refresh (default every 1 second).
  • The translog makes acknowledged writes survive crashes; a flush commits segments to disk and trims the translog.
  • Segments are immutable; deletes are markers and an update is a delete plus a new document.
  • Merges combine segments in the background and finally drop deleted documents.
  • Use refresh=wait_for sparingly, and raise refresh_interval for bulk loads.

Next: how fields are typed and stored in Mappings and Field Types.