Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Architecture and Operations

Integrations: ERP, PIM, OMS and Marketplaces

How a shop exchanges data with ERP, PIM, OMS and marketplaces: sync patterns, polling vs webhooks, field mapping, conflict resolution, rate limits and building a product feed.

A shop rarely lives alone. Product data may come from a PIM (product information management system), stock and invoices from an ERP (enterprise resource planning), shipments from a warehouse or an OMS (order management system), and the same catalog may be sold on marketplaces and advertised through product feeds. Each connection is a small distributed system with its own failure modes. Most integration bugs are not about HTTP; they are about deciding who owns which data and what happens when two systems disagree.

This lesson covers the patterns that make these connections reliable. They apply regardless of platform, and they lean on ideas from earlier lessons: webhooks, idempotency and the domain model.

The systems around the shop

System Typically owns Flows to the shop Flows from the shop
PIM Names, descriptions, attributes, images Product content Rarely anything
ERP Costs, stock, invoices, customers (accounting) Stock, prices, tax codes Orders, payments, refunds
OMS / WMS Order routing, picking, stock by location Shipment status, tracking Paid orders
Marketplace Its own listings and orders Orders, messages Listings, stock, prices
Ad / comparison channels Nothing Nothing Product feeds
flowchart LR
  PIM["PIM"] -->|"products"| SHOP["Shop"]
  ERP["ERP"] -->|"stock, prices"| SHOP
  SHOP -->|"paid orders"| OMS["OMS / warehouse"]
  OMS -->|"shipments"| SHOP
  SHOP -->|"orders, invoices"| ERP
  SHOP -->|"feed"| ADS["Google Merchant,<br/>comparison sites"]
  SHOP <-->|"listings, orders"| MP["Marketplace"]

Arrows show who sends what; every arrow is a place where data can be late, wrong or duplicated.

The first design decision for each kind of data is the system of record: the one system whose value wins. If the PIM owns product titles, nobody edits titles in the shop admin, or the next sync overwrites the edit. Write the ownership table down. Half of the "the sync is broken" tickets turn out to be two teams editing the same field.

Sync patterns

There are four common ways to move data, and most real integrations mix them.

Batch files. The ERP drops a CSV or XML file on an SFTP server every night; the shop imports it. Simple and old-fashioned, and still common. Weaknesses: latency of hours, and a half-uploaded file can be imported. Mitigate by uploading under a temporary name and renaming when complete, and by validating the whole file before applying any of it.

Polling. The shop asks, "What changed since my last sync?" every few minutes. It works against any API, but you waste calls when nothing changed, and you must store a cursor (a timestamp or a change token) so each poll only fetches new data. Use the source system's own updated_at or version, not your clock, and overlap windows slightly to avoid missing records written during the poll.

Webhooks. The source calls you when something happens. This is fast and cheap, but delivery is at-least-once, events can arrive out of order, and you will occasionally miss some. Treat a webhook as a hint: verify its signature, store the event, and, where correctness matters, fetch the current state from the API instead of trusting the payload alone.

Message queues and event streams. In larger setups, systems publish events to a broker and consumers process them at their own pace. You gain decoupling and replay; you take on operating the broker.

A robust design combines webhooks for speed with a periodic reconciliation poll that catches whatever the webhooks missed. Assume that any push channel will lose messages sometimes.

Mapping data between systems

Two systems never describe a product in the same way. Your integration layer needs an explicit mapping: which source field becomes which target field, with what transformation.

  • Identifiers. Keep a table of (system, external_id) -> internal_id. Never assume the ERP's item number is your product ID. Store external IDs on your records so you can find them again.
  • Units and formats. Prices as decimal strings versus integer minor units (money as integers), weights in grams or pounds, country names versus ISO codes, local time versus UTC.
  • Enumerations. The ERP's order status "07" must become your fulfilled. Map unknown values to an explicit error, never silently to a default.
  • Structure. One ERP item may become several variants, or several ERP lines may become one shop line. Define the rule once.
  • Validation at the border. Reject or quarantine rows with a missing SKU, negative stock or an unknown currency, and report them. A bad row should not stop the import of 9,999 good ones, and it should not disappear unnoticed either.

Keep the raw payload you received (at least for a while). When a customer asks why a price was wrong at 3 a.m., the raw record answers the question.

Conflicts and ordering

Conflicts happen when two writers change the same thing. Strategies, from simplest to most careful:

  1. Single owner. Only one system may write the field. This avoids almost all conflicts.
  2. Last write wins, based on a trustworthy timestamp or version number from the source. Cheap, but you can silently lose edits.
  3. Version checks. Every update carries the version it was based on; the receiver rejects stale updates. This is the same optimistic locking used for stock.
  4. Field-level merge. Accept changes to different fields from different sources, and apply the owner rule per field.
  5. Manual resolution. Park the conflict in a queue for a human when money or legal data is involved.

Stock deserves special care. If the ERP says 12 and the shop sold 3 since the last sync, overwriting the shop's count with 12 puts three sold units back on the shelf. Safer patterns: send deltas (movements) rather than absolute counts, or send absolute counts with a timestamp and let the shop re-apply the sales that happened after that moment. See inventory tracking.

Rate limits, retries and failures

External APIs limit how fast you may call them. Marketplaces often allow only a handful of requests per second, and your nightly full sync can exhaust the budget that live orders need.

  • Respect limits. Read the rate-limit headers or the documented quota, and throttle on your side with a token bucket. Reserve capacity for time-sensitive calls such as accepting an order.
  • Back off. On a 429 or a 5xx response, wait and retry with exponentially growing delays plus random jitter. Honor a Retry-After header when present.
  • Be idempotent. A retry must not create a second order or ship twice. Use idempotency keys where the API offers them, or check for the external ID before inserting.
  • Dead-letter queue. After a bounded number of attempts, move the failed message aside with its error, and alert. Do not retry forever.
  • Batch. Prefer bulk endpoints for catalog updates; one call with 100 items beats 100 calls.
  • Monitor lag. Measure the age of the oldest unsynced change. "Last successful sync: 14 hours ago" is a more useful alert than an error counter.

Product feeds and marketplaces

A product feed is a file or API that lists your products in a format an advertising or comparison channel understands. Google Merchant Center is the best-known example: shoppers see your products in shopping results, and the data comes from your feed. Each channel defines required attributes (an ID, title, description, link, image link, price, availability, condition, and for many products a brand and a GTIN, the barcode number), and rejects items that fail its rules.

Feed work is mostly careful data preparation:

  • Use the canonical product URL and a full-size image URL; the link must lead to a page showing the same price and availability.
  • Format prices the way the channel expects (for Google, a number followed by a currency code such as 19.99 USD).
  • Map your stock status to the channel's allowed values.
  • Escape text properly for the format (XML needs &amp; and friends), and strip HTML from descriptions.
  • Produce one row per variant when the channel wants sellable items, and group them with a shared parent ID.
  • Regenerate often enough that prices and availability do not drift; stale prices can get products disapproved.
Python Generate a product feed from catalog rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The generator does three things worth copying: it maps from an internal model rather than reading the database layout straight into the feed; it validates each row and reports rejections instead of failing the whole file; and it escapes text for the target format.

Marketplaces go further than feeds. You list products on the marketplace, they take orders, and you must keep stock and prices in step. Plan for: a different category and attribute taxonomy per marketplace, their own product identity (matching your SKU or GTIN to their catalog entry), fees and commission rules, strict response-time and cancellation metrics, and importing their orders into your OMS so that stock is shared. Overselling on a marketplace carries penalties, so keep a safety buffer (list 5 fewer units than you hold) and push stock changes quickly.

Python Applying stock updates with versions and deltas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive approach would have set stock to 12 and quietly resurrected three units that were already sold.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should each kind of data have a single system of record?

    If two systems both edit the same field, one will overwrite the other. A declared owner makes conflicts rare and explainable.
  2. 2. What is the best way to combine webhooks and polling?

    Webhooks are fast but at-least-once and sometimes lost. A slower reconciliation poll repairs any gaps.
  3. 3. The ERP says 12 units; the shop sold 3 since that snapshot. What should the shop show?

    Overwriting with the snapshot would resurrect units that were already sold. Apply the snapshot and then the later movements, or sync deltas.
  4. 4. A feed has 10,000 rows and 12 of them are invalid. What is the best behavior?

    Valid rows should still go out, but rejected rows must be reported so someone can fix the data.

Summary

  • Decide a system of record for each kind of data and write it down.
  • Sync with files, polling, webhooks or streams; pair fast push channels with reconciliation.
  • Map identifiers, units, enumerations and structure explicitly, validate at the border and keep raw payloads.
  • Resolve conflicts with ownership, versions or deltas; stock should move by movements, not blind overwrites.
  • Throttle, back off with jitter, make operations idempotent and park failures in a dead-letter queue.
  • Feeds need canonical URLs, correct price formats, valid availability and proper escaping; marketplaces add taxonomy, fees and overselling penalties.

The next lesson steps back to the biggest integration decision of all: Choosing a Platform: Build vs Buy.