Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 43 ยท Getting Started

Documents, Indexes and CRUD

Learn how Elasticsearch stores JSON documents in indexes, how _id and _source work, and how to index, get, update, delete and bulk-load data safely.

The unit of data in Elasticsearch is the document: one JSON object, such as a product, a blog post or a log line. Documents live in an index, a named collection of documents that are usually similar. If you know relational databases, an index is roughly like a table and a document is roughly like a row, but the comparison breaks down quickly: documents can be nested, fields can be missing, and there is no fixed schema you must declare first.

This lesson covers the four basic operations (index, get, update, delete), the metadata that comes back with every document, how versions protect you from lost updates, and how the bulk API loads many documents in one request. As before, the real requests are read-only; the JavaScript stand-in is an in-memory model with the same verbs, not the real engine.

What a document looks like

A document is JSON. You send it to an index, and Elasticsearch keeps your original JSON as the _source field. Beside it, every stored document has metadata fields that begin with an underscore:

JSON
{
  "_index": "books",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "title": "The Silent Harbor",
    "author": "Mara Lindqvist",
    "year": 2019,
    "tags": ["mystery", "sea"]
  }
}
  • _index is the index the document lives in.
  • _id identifies the document within the index. You can choose it, or let Elasticsearch generate a unique one.
  • _source is the original JSON you sent, returned exactly as stored. Search results show it by default.
  • _version starts at 1 and increases by one every time the document is changed.
  • _seq_no and _primary_term are counters from the shard that let you detect conflicting writes (more below).

An important point: _source is just a stored copy. What you can search is decided separately, by the mapping and the indexed terms (The Inverted Index and Mappings and Field Types).

Creating indexes

You can create an index explicitly:

JSON
PUT /books

You do not have to. If you index a document into an index that does not exist, Elasticsearch creates the index on the fly and guesses the field types (this is called dynamic mapping). That is convenient while learning and risky in production, where you normally create indexes with a deliberate mapping. Index names must be lowercase and cannot contain spaces or characters such as /, * or ?.

Index, get, update, delete

Index a document with your own id. PUT with an id creates the document, or replaces it if the id already exists:

JSON
PUT /books/_doc/1
{
  "title": "The Silent Harbor",
  "author": "Mara Lindqvist",
  "year": 2019
}

The response says "result": "created" the first time and "result": "updated" later, and _version goes up.

Let Elasticsearch choose the id. Use POST without an id when you do not care about the identifier (typical for logs):

JSON
POST /books/_doc
{ "title": "Notes from the Tide", "author": "Mara Lindqvist", "year": 2022 }

Create only if new. PUT /books/_create/1 (or _doc/1?op_type=create) fails with a 409 conflict if the id exists. Use it when overwriting would be a bug.

Get by id. This is a real-time lookup: it sees the latest version even before the next refresh (see Segments, Refresh and Near-Real-Time Search):

JSON
GET /books/_doc/1

You can fetch only the source (GET /books/_source/1) or a subset of fields with _source_includes.

Partial update. The update API changes only the fields you send:

JSON
POST /books/_update/1
{
  "doc": { "year": 2020, "tags": ["mystery"] }
}

Under the hood Elasticsearch does not edit the stored document. It reads the current _source, merges your fields into it, and indexes a whole new document, marking the old one as deleted. Updates are therefore "read, change, rewrite", which matters for performance on documents that change very often. Arrays are replaced, not merged: sending "tags": ["mystery"] removes the older tags.

Delete. Removing a document, or a whole index:

JSON
DELETE /books/_doc/1
DELETE /books

Deleting an index is immediate and permanent unless you have a snapshot. Many teams disable wildcard deletes with the setting action.destructive_requires_name for safety.

The four basic verbs

Index PUT or POST a JSON document.
Get Fetch one document by id.
Update Merge fields, rewrite the doc.
Delete Remove a document or an index.
1 / 2

Versions and conflicts

Imagine two web servers read the same product, each changes a different field, and each writes the whole document back. The second write silently erases the first change. This is a lost update.

Elasticsearch gives you optimistic concurrency control. Every write to a document gets a new _seq_no (increasing per shard) together with the shard's _primary_term. When you write, you can say "only do this if the document still has the sequence number I saw":

JSON
PUT /books/_doc/1?if_seq_no=3&if_primary_term=1
{ "title": "The Silent Harbor", "author": "Mara Lindqvist", "year": 2021 }

If someone else changed the document in the meantime, the request fails with 409 Conflict (version_conflict_engine_exception), and your code can re-read and retry. This is the recommended mechanism. The older version parameter still exists, but if_seq_no and if_primary_term are the modern way. The update API also has retry_on_conflict=3 to retry its read-merge-write cycle automatically.

Scripted updates, upserts and changing many documents

A plain doc update merges fields. When the new value depends on the old one, such as "add one to a counter" or "append a tag unless it is already there", send a script instead. The script runs inside the cluster on the current _source, so the read-change-write cycle needs no round trip to your application:

HTTP
POST /books/_update/1?retry_on_conflict=3
{
  "script": {
    "lang": "painless",
    "source": "ctx._source.views = (ctx._source.views == null ? 0 : ctx._source.views) + params.n",
    "params": { "n": 1 }
  }
}

Pass changing values through params instead of pasting them into the source. The script is then compiled once and cached, not once per value.

Upserts

An upsert means "update it if it exists, create it if it does not". There are two forms:

HTTP
POST /books/_update/99
{
  "doc": { "title": "A New Arrival", "views": 1 },
  "doc_as_upsert": true
}

POST /books/_update/99
{
  "script": { "source": "ctx._source.views += 1" },
  "upsert": { "title": "A New Arrival", "views": 1 }
}

With doc_as_upsert, the partial document is also used as the new document when the id is missing. With a script, the upsert document is stored as is if the id is missing and the script runs only when it exists. Upserts are ideal for counters and for event feeds where you do not know whether the record has been seen before.

Delete or update many documents by query

Sometimes you need to change or remove everything that matches a condition:

HTTP
POST /books/_update_by_query?conflicts=proceed
{
  "query": { "term": { "author.keyword": "M. Lindqvist" } },
  "script": { "source": "ctx._source.author = 'Mara Lindqvist'" }
}

POST /books/_delete_by_query?conflicts=proceed
{ "query": { "range": { "year": { "lt": 1900 } } } }

Both calls take a snapshot of the index when they start, then walk through the matching documents in batches. Each document is checked against the version it had in that snapshot. If a user changed it in the meantime, that single document is a version conflict. By default the whole request stops at the first conflict, and it has already changed some documents (there is no rollback). With conflicts=proceed the conflicting documents are counted in the response under version_conflicts and skipped, so you can run the request again later.

For big jobs add wait_for_completion=false. You get a task id back, and you follow it with GET /_tasks/<id>, or cancel it with POST /_tasks/<id>/_cancel. Useful companions are requests_per_second to throttle the load, and slices=auto to parallelize across shards. Deleting documents only marks them as deleted. Disk space comes back later when segments merge (see Segments, Refresh and Near-Real-Time Search), and deleting a whole index is far cheaper than deleting all its documents by query.

Optimistic concurrency in practice

The if_seq_no check from the previous section is a loop in real code: read the document with its _seq_no and _primary_term, compute the change, write it with those two values, and on a 409 start again. For simple changes the update API does this loop for you with retry_on_conflict. Use if_seq_no yourself when the change cannot be expressed as a script, for example when a person edited a form for a minute.

JS Upserts, scripted updates and a conflict-skipping update_by_query

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The stand-in keeps only the rules that matter here: a version number per document, a snapshot of sequence numbers, and the choice between stopping and skipping on a conflict. It does not have scripts in Painless, batches, throttling or tasks.

The bulk API

Sending one HTTP request per document is slow. The bulk API sends many operations in a single request. The body is NDJSON: newline-delimited JSON, where every line is a complete JSON object, and the body must end with a newline.

JSON
POST /_bulk
{ "index": { "_index": "books", "_id": "1" } }
{ "title": "The Silent Harbor", "year": 2019 }
{ "index": { "_index": "books", "_id": "2" } }
{ "title": "Notes from the Tide", "year": 2022 }
{ "update": { "_index": "books", "_id": "1" } }
{ "doc": { "year": 2020 } }
{ "delete": { "_index": "books", "_id": "2" } }

An action line (index, create, update, delete) is followed by a source line, except delete, which has none. The response contains "errors": true or false plus one result per item. Bulk is not a transaction: each item succeeds or fails alone, and the HTTP status is still 200 even when some items failed. Always check errors and inspect the failed items. A good batch size is often a few thousand documents or around 5-15 MB per request; measure on your own cluster. See Using Elasticsearch from Code for client helpers.

An in-memory document store

The stand-in below models index, create-only, get, partial update, version conflicts and delete. Its counters (_version, _seq_no) behave like the real ones for this simple one-shard case. Everything else (analysis, search, shards, refresh) is left out.

JS A document store with versions and conflicts

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Follow the counters. The first index creates version 1. Indexing the same id again is an updated with version 2, even if the JSON did not change (a plain index operation always rewrites). create on an existing id fails with 409. The partial update merges year and keeps title, making version 3. We note the _seq_no, let "another writer" change the document, and then our write with the old if_seq_no is rejected with a conflict, so it cannot erase the other change.

Parsing a bulk request

Bulk bodies are easy to get wrong because of the paired lines. This second stand-in parses an NDJSON string the way a server must: read an action line, then a source line when the action needs one, and report a per-item result.

JS Parsing NDJSON for the bulk API

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The bad document gets its own error entry, while the other items still succeed, and errors is true. That is exactly why a real bulk response must always be inspected, even though the HTTP status says 200. The parser is a toy that only shows the idea of per-item results.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the _source field?

    _source is a stored copy of your document. Searching uses separate index structures built from the mapping.
  2. 2. How does Elasticsearch perform a partial update?

    Documents are immutable inside segments. An update creates a new document and marks the old one deleted.
  3. 3. Your write used if_seq_no=7 but the document is now at seq_no 9. What happens?

    Optimistic concurrency control rejects the write so you can re-read and retry instead of losing the other change.
  4. 4. A bulk request returns HTTP 200 with "errors": true. What does that mean?

    Bulk is not transactional. Check each item in the response to find which operations failed.

Summary

  • A document is a JSON object stored in an index; _source keeps your original JSON.
  • PUT /index/_doc/id creates or replaces; POST /index/_doc generates an id; _create refuses overwrites.
  • GET by id is real-time; _update merges fields but rewrites the whole document.
  • _version, _seq_no and _primary_term support optimistic concurrency with if_seq_no and if_primary_term.
  • The bulk API takes NDJSON, is not transactional, and needs its errors flag and items checked.
  • The stand-ins model the verbs and counters only, not the real storage engine.

Next: The Inverted Index.