Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 28 ยท Projects

Project: Product Sync Tool

Build a supplier-feed product sync for the BigCommerce Catalog API: validate rows, plan creates and updates, batch writes, retry on rate limits and stay idempotent.

Almost every merchant with a supplier has the same chore: a spreadsheet or feed arrives with SKUs, names, prices and stock, and the store must match it. Done by hand it is slow and error-prone; done badly by a script it creates duplicates, overwrites good data and hammers the API until it is refused. In this project you will build the core of a product sync tool that does it properly.

The real tool would call the BigCommerce Catalog API with a store token; those calls are shown read-only. Everything you can run here uses a fake in-memory API that behaves like a simplified Catalog API, with batches capped at ten items and an occasional rate-limit response. The design is the real lesson: validate, plan, batch, retry, and make a second run change nothing. It builds on The Catalog API (V3), Orders, Customers and the V2/V3 Split for retries and Idempotency.

Requirements

Write down what "correct" means before coding:

  1. The supplier feed is the source of truth for name, price and stock of products whose SKU appears in it.
  2. A SKU that exists in the store is updated only when something actually differs. A SKU that does not exist is created.
  3. Bad rows (missing SKU, negative price, non-numeric stock) are reported and skipped; they never stop the run.
  4. Writes are batched at the platform's maximum size.
  5. Rate limits and temporary errors are retried with backoff; permanent errors are reported.
  6. Running the tool twice with the same feed makes no writes the second time.
  7. The tool never deletes products. Removal is a business decision, not a sync side effect.
flowchart LR
    F["Supplier feed"] --> P["Parse and<br/>validate"]
    P --> E["Fetch existing<br/>by SKU (chunks)"]
    E --> D["Diff:<br/>create / update / same"]
    D --> B["Batch writes<br/>(max 10)"]
    B --> R["Retry 429<br/>with backoff"]
    R --> Rep["Report"]

The pipeline: nothing is written until the plan is known.

Step 1: parse, validate and normalize

Never let raw feed data reach the API. Parse each line, trim and normalize it, and collect errors with their line numbers so the merchant can fix the source. Keep money in integer cents inside your program (Money as Integers) and convert to the API's decimal price only when sending. Normalize SKUs (trim, upper case) because "tote-01 " and "TOTE-01" are the same product to a human and different strings to a computer.

The first stand-in parses a small CSV feed and produces valid rows plus an error list.

Node.js Parse and validate a supplier feed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Three valid rows come through, and five problems are reported with line numbers, including the repeated BTL-04 (the first line wins). Look at the last problem: the supplier wrote the price 19,90 with a decimal comma inside quotes, and the naive split(",") cut it in half. The parser rejects the row instead of guessing, which is the safe outcome. A real tool needs a proper CSV parser for quoted fields, and perhaps a setting for the supplier's number format.

Step 2: plan before you write

Fetching existing products one SKU at a time would be an N+1 disaster. The Catalog API lets you filter by a list of SKUs, so fetch in chunks and build a map. The plan compares the feed row with the stored product and labels each row create, update or same. Because writes happen only for create and update, a second run produces an empty plan, and that is what idempotent means here: the end state depends on the feed, not on how many times you ran the job.

Real calls look like this (read-only):

HTTP
GET /v3/catalog/products?sku:in=TOTE-01,MUG-02,APR-03&include_fields=id,sku,name,price,inventory_level
X-Auth-Token: {token}

POST /v3/catalog/products          (create)
PUT  /v3/catalog/products          (batch update, each item has an id)

Stock on a product without variants is the product's inventory_level (when inventory tracking is set to product level); products with variants track stock per variant. Check the current reference for how your catalog tracks it before syncing stock, because writing the wrong field is a quiet way to break a store.

Step 3: batches and retries

Writes go in chunks of the batch limit. Each request can fail as a whole (a 429 Too Many Requests or a 5xx) or in part (individual items rejected). The whole-request failures are retryable: wait, then try again, increasing the wait each time, and give up after a few attempts. Item-level validation errors are not retryable; report them. The retry loop respects what the server says: a Retry-After-style hint is better than guessing, and it is wise to add a little random jitter in production so many clients do not retry in lockstep. Here the wait is recorded but not performed, so the demonstration is instant and repeatable.

Step 4: the whole run

The next stand-in is the complete tool. It contains a fake API with an in-memory store, capped batches, a SKU filter, and a deliberate 429 on the second write request. The sync runs twice on the same feed. The first run creates and updates; the second must do nothing.

Node.js Run the full sync twice against a fake API

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Run 1 plans 21 creates and two updates, with SKU-01 untouched. Creates go out in three batches (ten, ten and one); the second write request meets the fake 429, waits 500 ms (recorded, not slept), and succeeds on the retry. Run 2 plans nothing, sends no write requests and leaves 24 products, which is the idempotency requirement met.

Two limits of the stand-in are worth stating plainly. The real API has its own error formats, per-item results and rate-limit headers, and the real store may have products the feed knows nothing about. Also, if two sync jobs ran at the same time, they could both decide to create the same SKU; real tools run one job at a time (a lock or a queue) and treat a "duplicate" error on create as a signal to re-plan.

Taking it further

A production version would add:

  • Dry-run mode that prints the plan and exits.
  • Stock handling by variant, plus an "out of stock" rule when the feed drops a SKU (hide or zero rather than delete).
  • A report saved after every run (counts, skipped rows, failures) so the merchant sees what happened.
  • Scheduling with a queue, and an alert when a run fails or changes an unusual number of products. A feed that suddenly changes every price by 90% should stop and ask a human.
  • Safeguards such as a maximum percentage of products allowed to change in one run.
  • Secrets from the environment, with a token that has catalog scope only (API Accounts, Tokens and Scopes).
  • Tests: unit tests for parsing and diffing, and the mock API approach from Testing Apps and Themes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the tool build a plan (create, update, same) before writing anything?

    Comparing first means only real differences are written, so repeated runs end in the same state and use few API calls.
  2. 2. Which failure should be retried with backoff?

    Rate limits and temporary server errors are transient. Data problems will fail the same way every time, so report them instead.
  3. 3. Why fetch existing products with a list of SKUs instead of one request per SKU?

    One filtered request per chunk replaces many single requests, which is faster and uses less of the rate limit.
  4. 4. Why does the tool never delete products that are missing from the feed?

    A truncated feed would otherwise wipe the catalog. Hide or flag products and let a person decide.

Summary

  • A good sync validates input, plans changes, batches writes, retries transient failures and is idempotent.
  • Normalize SKUs, keep money in integer cents and reject ambiguous numbers instead of guessing.
  • Fetch existing records in filtered chunks and compare field by field.
  • Batch to the platform's cap; retry only retryable errors with growing waits.
  • Report skipped rows with line numbers, and add safeguards before running on a live store.

Next: Project: Order Webhook Notifier App