Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 43 ยท Working from Code

Using Elasticsearch from Code

Use Elasticsearch from your app: official clients, bulk indexing, retries, mapping errors, and zero-downtime reindex with aliases, plus a Laravel Scout note.

Everything you have done so far with curl and Dev Tools is plain HTTP and JSON. Your application does the same thing, usually through an official client library that builds the requests, handles connections, and decodes responses. The calls you already know (index, get, search, bulk) have the same names and bodies in almost every language.

This lesson covers what is different in real programs: bulk indexing with partial failures, retries, mapping errors, and the alias-and-reindex pattern that lets you change an index without downtime. Because no server can run on this site, the client code is read-only, and the runnable examples use a small mock client written in Node.js.

Official clients

Elastic maintains clients for the main languages. The names below are the package names at the time of writing; check the current documentation before you install, because major versions change the API.

Language Package
JavaScript / Node.js @elastic/elasticsearch
Python elasticsearch
PHP elasticsearch/elasticsearch
Java co.elastic.clients:elasticsearch-java

Use a client whose major version matches your cluster's major version. Clients for version 8 and 9 talk to the same kind of REST API; they differ mainly in typed helpers and compatibility headers.

A typical connection reads the URL and credentials from the environment, never from source code:

JavaScript
// Node.js (read-only: needs a running cluster and an npm package)
import { Client } from '@elastic/elasticsearch';

const client = new Client({
  node: process.env.ES_URL,                    // https://localhost:9200
  auth: { apiKey: process.env.ES_API_KEY },    // see the security lesson
});

const result = await client.search({
  index: 'products',
  query: { match: { name: 'trail shoe' } },
  size: 10,
});

for (const hit of result.hits.hits) {
  console.log(hit._id, hit._score, hit._source.name);
}
Python
# Python (read-only)
import os
from elasticsearch import Elasticsearch

es = Elasticsearch(os.environ["ES_URL"], api_key=os.environ["ES_API_KEY"])

result = es.search(index="products", query={"match": {"name": "trail shoe"}}, size=10)
for hit in result["hits"]["hits"]:
    print(hit["_id"], hit["_score"], hit["_source"]["name"])
PHP
<?php
// PHP (read-only)
use Elastic\Elasticsearch\ClientBuilder;

$client = ClientBuilder::create()
    ->setHosts([getenv('ES_URL')])
    ->setApiKey(getenv('ES_API_KEY'))
    ->build();

$response = $client->search([
    'index' => 'products',
    'body'  => ['query' => ['match' => ['name' => 'trail shoe']], 'size' => 10],
]);

foreach ($response['hits']['hits'] as $hit) {
    echo $hit['_id'], ' ', $hit['_source']['name'], "\n";
}

Good habits for any client:

  • Create one client per process and reuse it. It keeps connections alive and pools them.
  • Set sensible timeouts (request and connect) so a slow cluster does not hang your web workers.
  • Let the client retry on connection errors and on 429/502/503/504; keep the retry count small and use backoff.
  • Do not build query JSON by gluing user text into strings. Pass user input as values inside the structure (more in security and operations).

A client does four jobs

Connects Pools and reuses connections.
Serializes Turns objects into JSON requests.
Retries Handles node failures and 429s.
Helpers Bulk, scroll and reindex helpers.
1 / 2

Bulk indexing

Sending one HTTP request per document is slow. The bulk API accepts many operations in one request. The body is newline-delimited JSON: an action line, then (for index and create) a source line, each ending in a newline:

HTTP
POST /_bulk
{ "index":  { "_index": "products", "_id": "1" } }
{ "name": "Trail shoe", "price": 89.0 }
{ "index":  { "_index": "products", "_id": "2" } }
{ "name": "Rain jacket", "price": "cheap" }
{ "delete": { "_index": "products", "_id": "9" } }

The most important thing to know: a bulk request is not a transaction. The HTTP status is usually 200 even if some items failed. The response has a top-level errors flag and an items array with one result per operation, in the same order:

JSON
{
  "took": 8,
  "errors": true,
  "items": [
    { "index":  { "_id": "1", "status": 201, "result": "created" } },
    { "index":  { "_id": "2", "status": 400,
                  "error": { "type": "document_parsing_exception",
                             "reason": "failed to parse field [price] of type [float]" } } },
    { "delete": { "_id": "9", "status": 404, "result": "not_found" } }
  ]
}

So your code must always check errors, walk items, and decide what to do with each failure. All three clients ship a bulk helper that batches, streams and reports failures for you (helpers.bulk in Python and JavaScript). Typical batch sizes are a few thousand small documents or about 5 to 15 MB; tune by measuring.

Which failures should be retried?

  • 429 Too Many Requests (the write queue is full) or a 503: retry later with exponential backoff. The cluster is busy, not unhappy with your data.
  • 400 mapping or parsing errors: never retry as-is. The document does not fit the mapping. Log it, send it to a dead-letter store, fix the data or the mapping.
  • 409 version_conflict_engine_exception: someone else changed the document first. Re-read and decide, or use retry_on_conflict for updates.
  • Network timeouts: the request may have succeeded. Retrying an index with an explicit _id is safe (it overwrites the same document). Retrying a create with auto-generated IDs may produce a duplicate.
Node.js A mock client with bulk and partial failures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The output shows the three outcomes in one response: success (201), a permanent mapping failure (400, which is not retried), and a temporary rejection (429, retried once and then accepted). This is the shape of logic every production bulk loader needs.

Mapping errors

Two kinds of mapping problems show up in code:

  • Parsing failure (document_parsing_exception, a 400): the value cannot be converted to the mapped type, such as "cheap" into a number. Fix the data, or use ignore_malformed on that field if losing a bad value is acceptable.
  • Mapping conflict: a field was mapped as long and you now send text, or you want to change an existing field's type. Elasticsearch does not allow changing the type of an existing field. You create a new index with the right mapping and reindex into it.

Zero-downtime changes: aliases and reindex

Your application should never hard-code a concrete index name like products-v1. Instead, it searches and writes through an alias, a pointer to one or more indexes. When the mapping must change, you build a new index and flip the alias in one atomic step:

HTTP
# 1. The app uses the alias "products", currently pointing at products-v1
PUT /products-v2
{ "mappings": { "properties": { "price": { "type": "scaled_float", "scaling_factor": 100 } } } }

# 2. Copy the data (runs as a background task)
POST /_reindex?wait_for_completion=false
{ "source": { "index": "products-v1" }, "dest": { "index": "products-v2" } }

# 3. Swap atomically: searches see either all-old or all-new
POST /_aliases
{ "actions": [
  { "remove": { "index": "products-v1", "alias": "products" } },
  { "add":    { "index": "products-v2", "alias": "products" } }
] }

While step 2 runs, writes to the old index continue, so the new index can miss them. Typical solutions: pause writes briefly, run a second incremental reindex using a timestamp filter, or have your application write to both indexes during the migration. Keep products-v1 for a while as a rollback option and delete it later.

Node.js Switching an alias atomically

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The application code calls search('products') both times and never notices the switch. That is the whole point of the alias.

Laravel Scout note

If you build with Laravel, Scout gives models a Searchable trait and engine drivers. Scout ships drivers for Meilisearch, Algolia, Typesense and a database engine; Elasticsearch is available through community packages, whose features and Elasticsearch version support vary, so check that a package is maintained before relying on it. Scout is convenient for simple "search this model" needs. For custom mappings, analyzers, aggregations and facets, using the official PHP client directly (or a thin service class around it) is usually clearer. Either way, the ideas in this lesson apply: use an alias, index in bulk, and handle partial failures.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A bulk request returns HTTP 200 with errors: true. What does this mean?

    Bulk is not transactional. Successful items are stored even if others fail, and the response lists the status of each item in order.
  2. 2. Which bulk item failure is worth retrying later with backoff?

    A 429 means the cluster was too busy right now. A 400 means the document itself does not fit the mapping, so retrying the same data will fail again.
  3. 3. How do you change the type of an existing field with no downtime for the application?

    Field types cannot be changed in place. A new index plus an atomic alias swap lets the app keep using the same alias name.
  4. 4. Why should application code use an alias instead of a concrete index name?

    An alias is only a pointer. You can repoint it atomically to a new index.

Summary

  • Use an official client of the matching major version, reuse one instance, set timeouts, and keep credentials in the environment.
  • Bulk requests are newline-delimited JSON and not transactional: always read errors and items.
  • Retry temporary failures (429, 503, timeouts) with backoff; never retry mapping errors unchanged.
  • Field types cannot be changed in place: create a new index, reindex, and swap an alias atomically.
  • Laravel Scout is handy for simple model search; use the PHP client for mappings, analyzers and aggregations.

Next: locking it down and keeping it healthy in Security, Backups and Operations.