Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 43 ยท Mappings and Data Modeling

Data Modeling and Denormalization

Model Elasticsearch data for the queries you run, not for tables: denormalization, one-to-many choices, runtime fields, update costs and tenant layouts.

In a relational database you normalize: each fact is stored once and tables are joined at query time. Elasticsearch has no general joins, and it is fast precisely because every search reads self-contained documents. So the habit flips. You design documents around the questions you will ask, and you accept some duplication to make those questions cheap.

This lesson gives you a way to think about it: start from queries, pick how to represent one-to-many data, decide what to compute at index time and what at query time (runtime fields), weigh update costs, and choose how to separate tenants. The requests are read-only; a JavaScript stand-in measures what denormalization costs when data changes.

Model for the question

List the queries Search, filter, sort and aggregate needs first.
One doc per hit A result row should be one document.

If the screen shows a product, index products.

1 / 2

Start from the query, not the schema

Suppose a shop has products, brands and categories tables. The page you build shows product cards with the brand name and category, filtered by brand and category, sorted by price. In SQL you would join three tables. In Elasticsearch you index one product document that already contains what the page needs:

JSON
{
  "sku": "TS-100",
  "name": "Trail shoe",
  "price": 59.0,
  "brand": { "id": "acme", "name": "Acme" },
  "category": ["Shoes", "Running"],
  "in_stock": true
}

A search returns finished cards in one request, with no join and no second lookup. The brand name is duplicated across every Acme product. That is the deal.

One-to-many: your choices

When one thing has many related things (a product with reviews, an order with lines), you have four choices, in rough order of preference:

  1. Keep it flat. Store only what queries need (a review count and average rating on the product). Cheapest and fastest.
  2. Embed as an object array when you never query combinations inside the list. Cheap, but the pairing is lost (see Objects, Arrays and Nested Fields).
  3. Embed as nested when you must match several fields of the same element. Correct, costs more at index and query time.
  4. Separate documents in their own index (and possibly a join field) when the children are numerous or change independently. You then run two queries from your application, or use join.

Often the right design is a mix: an orders index with one document per order and nested lines, plus a separate order_lines index if analysts need to aggregate millions of individual lines.

What if you need a join anyway?

Elasticsearch offers join fields and terms lookups, and you can do an application-side join: query index A, take the IDs, query index B with terms. These work for modest cases but add latency and complexity. If your core use case is many joins with strict consistency, a relational database should stay the source of truth, with Elasticsearch as a read-optimized search copy. Many systems do exactly that.

Runtime fields: computing at query time

Sometimes you want a value that is not stored, or you only decided you need it after the data is indexed. A runtime field is calculated when a query runs, using a script, so you can add it without reindexing.

HTTP
PUT /products/_mapping
{
  "runtime": {
    "price_with_tax": {
      "type": "double",
      "script": { "source": "emit(doc['price'].value * 1.2)" }
    }
  }
}

GET /products/_search
{
  "query": { "range": { "price_with_tax": { "gte": 50 } } }
}

The script language is Painless, and emit returns the computed value. Runtime fields are flexible and great for exploring or fixing a modeling gap. They are slower than indexed fields because the script runs for each candidate document, so once a runtime field proves useful, move it into the real mapping and fill it at index time (or with a reindex).

Update cost: the price of duplication

Denormalization moves work from reads to writes. If Acme renames itself, every product with that brand must be updated, and each update rewrites a whole document (a delete plus a new document, as in Segments, Refresh and Near-Real-Time Search). That is fine for data that changes rarely (brand names, categories) and painful for data that changes constantly (a live view counter on every document).

Rules of thumb:

  • Duplicate data that is read often and changes rarely.
  • Keep fast-changing counters in a separate index or compute them elsewhere.
  • Use _update_by_query or reindex for bulk corrections, and design the pipeline that feeds Elasticsearch to push changes in bulk.
JS Denormalization: cheap reads, costly updates

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With four products the difference is tiny. With four million Acme products, the denormalized rename is a large reindexing job, while each read stays a single lookup. Decide by comparing how often you read and how often you rename.

Computing at index time versus query time

JS Index-time field versus runtime field

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Same answer, different cost profile: the runtime version ran the script once per document per query.

Multi-tenant layouts

If your application serves many customers (tenants), you can organize data in two main ways:

  • Index per tenant. Strong isolation, per-tenant settings, easy deletion of a tenant, simple security by index name. But each index has shards that use memory, so thousands of small tenants create too many shards and an overloaded cluster.
  • Shared index with a tenant_id filter. Few indexes and shards, efficient for many small tenants. Every query must include a tenant_id filter, ideally enforced by a filtered alias or by document-level security so a coding mistake cannot leak data. Setting routing to the tenant ID keeps a tenant's documents on one shard and makes queries cheaper.

A common compromise: a shared index for the many small tenants and dedicated indexes for the few very large ones.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the guiding principle when modeling data for Elasticsearch?

    There are no cheap joins, so documents are shaped to answer your searches directly, even if that duplicates some data.
  2. 2. Which data is a good candidate for denormalizing into every document?

    Duplicate what is read often and changes rarely, because every change means rewriting all copies.
  3. 3. What is the main downside of a runtime field compared with an indexed field?

    Runtime fields need no reindex, but the value is computed per document on every query.
  4. 4. Why can an index per tenant fail at large scale?

    Each shard has overhead, so too many tiny indexes overload the cluster. A shared index with a tenant filter scales better for many small tenants.

Summary

  • Model documents around queries, so a search hit is one self-contained document.
  • For one-to-many data, prefer flat summaries, then object arrays, then nested, then separate documents or join.
  • Denormalization speeds reads but makes updates rewrite every copy; use it for slow-changing data.
  • Runtime fields add computed values without a reindex, at the cost of query speed; promote them to real fields when they prove useful.
  • Choose index per tenant for isolation and few large tenants, a shared index with a filter for many small ones.

Next: Query DSL Basics.