Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 55 ยท Webhooks

Reliable Webhook Handlers

Build Shopify webhook handlers that survive retries, duplicates and out-of-order events: answer fast, queue the work, de-duplicate by webhook id and reconcile with scheduled jobs.

A verified webhook is only half of the job. The other half is surviving real-world delivery. Shopify, like every webhook sender, promises at-least-once delivery: an event will arrive one or more times, but not necessarily once, not necessarily in order, and in rare cases not at all. If your handler assumes "exactly once, in order, always", it will eventually double-charge a usage meter, restore stock that was already restored, or silently miss an uninstall.

This lesson teaches the five habits of a reliable handler: authenticate, acknowledge quickly, de-duplicate, process in a queue with safe retries, and reconcile on a schedule to repair what slipped through. The first habit was covered in Verifying Webhook Signatures. The general theory of repeat-safe operations lives in Idempotency; here we apply it to Shopify's headers and timing.

What Shopify expects from you

Shopify waits only a short time for an answer. As a rule of thumb, treat it as a few seconds (the documentation currently says about five). If your endpoint does not return a 2xx status in time, Shopify considers the delivery failed and retries it on a schedule that stretches over several hours. If failures continue, Shopify may stop delivering to that subscription altogether. Check the current limits in the webhook documentation before you rely on exact numbers; the principle is stable:

  • Do not call slow APIs, send emails or run big queries inside the request.
  • Return 200 as soon as the delivery is verified and safely stored.
  • Do the real work afterwards.
flowchart LR
    S["Shopify"] -->|"POST + headers"| E["Endpoint"]
    E -->|"1 verify HMAC"| V{"Valid?"}
    V -->|"no"| R401["401"]
    V -->|"yes"| D{"Webhook id<br/>seen before?"}
    D -->|"yes"| R200a["200 (ignore)"]
    D -->|"no"| Q["Store id and enqueue job"]
    Q --> R200b["200 OK"]
    Q -.-> W["Worker processes job<br/>(retries on failure)"]

The request only verifies, records and queues. The slow work happens in a worker.

Idempotency with the webhook id

An operation is idempotent when doing it twice has the same effect as doing it once. Shopify helps you here: every delivery has an X-Shopify-Webhook-Id header, and a retry of the same delivery carries the same value. Shopify also sends X-Shopify-Event-Id, which identifies the underlying event; the same event can reach several subscriptions, so check the current documentation to decide which of the two fits your design. For simple per-endpoint de-duplication, the webhook id is the natural key.

Store the id in a table with a unique constraint, in the same database transaction that creates the job. A plain "if not seen then insert" in application code is racy: two retries can arrive at the same instant on two servers. The database constraint decides who wins.

TypeScript
// read-only sketch (Prisma-style)
try {
  await db.$transaction([
    db.webhookReceipt.create({ data: { webhookId, shop, topic } }), // unique(webhookId)
    db.job.create({ data: { webhookId, shop, topic, payload } }),
  ]);
} catch (error) {
  if (isUniqueViolation(error)) return new Response(); // already queued: still answer 200
  throw error;
}
return new Response();

Note that a duplicate still gets a 200. If you returned an error for it, Shopify would just keep retrying.

The second kind of idempotency is in the work itself. Compare two handlers for an inventory event:

  • Delta style: "add 2 to stock". Running it twice gives the wrong number.
  • State style: "set stock to 8". Running it twice gives the same number.

Prefer state-style updates whenever the payload contains the new value. When you must apply a delta (a counter or a usage charge), guard it with the webhook id.

Node.js Acknowledge fast, de-duplicate, process in a queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at the log: the failed job is retried and finishes on its second attempt, and the duplicate of w-001 never reached the worker. If a job fails again and again, move it to a dead-letter list (or table) so a human can look at it, instead of retrying forever.

Ordering: do not trust arrival order

Events are generated in order, but network delivery, retries and parallel workers can reorder them. An orders/updated event for 10:05 may arrive after the one for 10:07. If your handler blindly overwrites, the older data wins and you show a stale order.

Use one of two defences:

  1. Compare timestamps. Payloads usually carry updated_at. Store the last updated_at you applied per record and ignore any event that is not newer. (Shopify also sends X-Shopify-Triggered-At, which tells you when the event was generated.)
  2. Treat the webhook as a signal, not a source. Use the event only to learn that something changed, then fetch the current state from the Admin API and save that. The extra request is the price of always seeing the latest data. With many events per second, remember the budget in Rate Limits and Query Cost.

Reconciliation: a safety net

Even a well-built system can miss an event: your server was down for longer than the retry window, a deploy lost a queued job, or a subscription was removed after failures. So never make webhooks your only way to learn about changes. Add a reconciliation job that runs on a schedule (for example hourly or nightly):

  1. Ask the Admin API for records changed since the last successful run, usually with a search filter such as updated_at:>'2026-01-15T00:00:00Z' (see Pagination with Cursors).
  2. For each record, apply the same idempotent, timestamp-checked logic that the webhook handler uses.
  3. Remember the time you started the run, minus a small safety margin, for the next one.

For very large shops, use Bulk Operations so the job does not exhaust your rate limit.

Node.js Out-of-order events and a reconciliation pass

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The late open event is skipped, the duplicate is skipped, and the reconciliation pass fills the gap for order 2 using the same apply function. Reusing one code path for webhooks and reconciliation is deliberate: there is only one place where bugs can hide.

Practical checklist

  • Verify first, always. Unsigned requests get 401 and touch nothing.
  • Answer in well under your time budget. Put only verification, de-duplication and queuing in the request.
  • Make the receipt and the job one transaction. Otherwise a crash between the two loses the event or runs it twice.
  • Make workers safe to retry. Use state-style updates, or guard deltas with the webhook id.
  • Cap retries and keep a dead-letter list. Alert when it is not empty.
  • Subscribe only to topics you handle. Fewer topics means less traffic and less code.
  • Reconcile on a schedule. It is the only defence against events you never received.
  • Handle app/uninstalled by removing the shop's sessions and scheduling data cleanup; it also stops any reconciliation job for that shop.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should a webhook endpoint return 200 quickly and do the real work in a background job?

    Shopify waits only a few seconds. A slow handler looks like a failed delivery, so Shopify retries it (creating duplicates) and may eventually stop delivering. Verify, store, enqueue and answer; process later.
  2. 2. Where should the unique webhook id be enforced to avoid a race between two simultaneous retries?

    Application-level checks can race across processes and servers. A unique constraint lets the database decide a single winner, and writing the job in the same transaction prevents lost or doubled work.
  3. 3. Which update is naturally safe to repeat?

    A state-style update ('set to 8') gives the same result however many times it runs. Deltas and appends need an additional guard such as the webhook id.
  4. 4. An orders/updated event for 10:05 arrives after the one for 10:07. What protects you from overwriting newer data?

    Arrival order is not guaranteed. Storing the last applied updated_at per record and skipping older events keeps the newest state, whatever the delivery order.
  5. 5. Why add a scheduled reconciliation job when you already have webhooks?

    Delivery is best-effort. A periodic query for records changed since the last run, applied with the same idempotent logic, closes the gaps.

Summary

  • Webhook delivery is at-least-once, can be reordered and can occasionally fail; design for all three.
  • Verify, de-duplicate by X-Shopify-Webhook-Id (unique constraint), enqueue and return 200 fast.
  • Make workers retry-safe: prefer state-style updates, cap attempts and keep a dead-letter list.
  • Guard against stale events with updated_at, or use the event only as a signal and fetch fresh data.
  • Run a reconciliation job that reuses the same apply logic to heal missed events.

Next: App Extensions Overview