Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 28 ยท Jobs, Content and Operations

Jobs, Feeds and Import/Export

Run batch work on Salesforce B2C Commerce: job steps, custom steptypes.json steps, schedules, catalog, price and inventory imports, XML feeds, error handling and idempotent imports.

Not everything happens in a shopper's request. Prices arrive nightly from an ERP, a catalog is refreshed by a merchandising team, orders are exported to a warehouse, old data is cleaned up. These tasks are batch jobs: they run on a schedule or on demand, process many records, and must not be limited by the short timeouts of storefront requests.

Commerce Cloud has a built-in job framework. A job is an ordered list of steps; each step is either a built-in import/export step or a custom script you write. This lesson covers how jobs are defined and scheduled, how to write a custom step, what the XML feeds look like, and, most importantly, how to make imports safe to run twice. Nothing here runs on this site (it needs a real instance), but the final stand-in is a Node program that streams a feed and applies it idempotently.

Anatomy of a job

In Business Manager, under Administration > Operations > Jobs, a job has:

  • Steps run in sequence (and, in newer configurations, in flows that can run in parallel). Each step has a type, parameters and an error policy: stop the job on error, or continue.
  • Schedule and triggers: run once at a time, repeat on an interval or on a calendar, or start only by hand or by API. A job can also be started by another job's end.
  • Scope: some steps run in organization context (global data such as catalogs) and some in site context (site-specific data such as orders). Choose the scope when you add the step, and the job runs in all selected sites or only some.
  • Status and logs: every run has a status (OK, ERROR, warning) and a log file. A failed step is visible in Business Manager and can notify by email.

Platform-provided steps cover the common cases: importing and exporting catalogs, price books, inventory lists, customers, orders, content and site data; indexing and cleanup tasks; and generic file operations like upload, download and archive. Which step names exist and which parameters they take are in the current documentation and in the step picker of Business Manager.

flowchart LR
    T["Trigger<br/>(schedule or manual)"] --> J["Job"]
    J --> S1["Step 1<br/>Download file (SFTP)"]
    S1 --> S2["Step 2<br/>Import prices"]
    S2 --> S3["Step 3<br/>Custom: validate & report"]
    S3 --> R["Status + log"]
    S2 -.->|"error policy"| X["Stop or continue"]

A job is a pipeline of steps; the error policy decides what a failing step does to the rest.

Files and the IMPEX folder

Jobs exchange files through the instance's IMPEX area, a storage location reachable by WebDAV and by the Script API (dw.io.File.IMPEX). By convention, inbound files go to src/... subfolders, outbound ones to dst/..., and logs to log/. A typical overnight flow is:

  1. An external system (or a download step) puts prices-2026-10-30.xml into IMPEX/src/prices/.
  2. The import step reads it, applies the changes, and the job archives the file.
  3. Failure details go to the job log, and a notification goes to the owners.

Writing a custom job step

When no built-in step fits, you add your own. You declare it in steptypes.json at the root of the cartridge:

JSON
{
  "step-types": {
    "script-module-step": [
      {
        "@type-id": "custom.ExportOrders",
        "@supports-parallel-execution": "false",
        "@supports-site-context": "true",
        "@supports-organization-context": "false",
        "description": "Exports new orders to the warehouse.",
        "module": "int_warehouse/cartridge/scripts/steps/exportOrders.js",
        "function": "execute",
        "parameters": {
          "parameters": [
            {
              "@name": "BatchSize",
              "@type": "long",
              "@required": "false",
              "description": "How many orders per run."
            }
          ]
        },
        "status-codes": {
          "status": [
            { "@code": "ERROR", "description": "Used when the export failed." },
            { "@code": "OK", "description": "Used when the export finished." }
          ]
        }
      }
    ]
  }
}

The module path starts with the cartridge name; the cartridge must be on the cartridge path that the job's scope uses, or the step will not be found. The script exports the named function and returns a Status:

JavaScript
// exportOrders.js (read-only: needs the real platform)
'use strict';

var Status = require('dw/system/Status');
var Logger = require('dw/system/Logger');

exports.execute = function (parameters, stepExecution) {
    var batchSize = parameters.BatchSize || 100;
    var log = Logger.getLogger('warehouse', 'export');
    // ... query orders, call the warehouse service, update export status ...
    log.info('Exported {0} orders', batchSize);
    return new Status(Status.OK);
};

For big data sets, use a chunk-oriented step (chunk-script-module-step) instead. You provide functions to read one item, process it and write a batch; the framework calls them in chunks of a size you configure, commits after each chunk, and can resume cleanly. This keeps memory low and transactions short. In a plain script step, read through iterators, close them, and never load millions of objects into an array.

If a step needs to call a partner system, use the service framework from Service Framework and Third-Party Integrations, with a timeout suited to batch work.

What feeds look like

Imports and exports use XML documents with a namespace and a schema. Here is a heavily shortened catalog fragment, to show the shape (real files include many more optional elements):

XML
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.demandware.com/xml/impex/catalog/2006-10-31" catalog-id="storefront-catalog">
    <product product-id="SHIRT-001">
        <display-name xml:lang="x-default">Linen shirt</display-name>
        <online-flag>true</online-flag>
    </product>
    <product product-id="OLD-999" mode="delete"/>
</catalog>

The mode attribute (or the step's import mode) decides what happens to existing records. Merge adds and updates only the fields present, replace overwrites the whole record, and delete removes it. Know which one your job uses; a feed that omits fields behaves very differently under merge and replace.

Price books and inventory lists have their own formats, and custom feeds (a CSV from the ERP, JSON from a partner) are read in a custom step, with dw.io.FileReader, CSVStreamReader or XMLStreamReader (a streaming parser that reads one element at a time instead of loading the whole file).

Error handling that survives production

A good job is boring when something goes wrong. Plan these from the beginning:

  • Validate early. Check the file exists, is non-empty and has the right root element before changing anything.
  • Separate bad records from bad runs. One malformed product should be logged and skipped, not abort 200,000 good ones; but 40% bad records means something is wrong with the file, so fail the run (set an error threshold).
  • Use meaningful statuses. Return OK, a warning-style status for "finished with skipped records", or ERROR, and put counts in the log (read, changed, skipped, failed).
  • Archive, do not delete. Move processed files to an archive folder so you can replay them.
  • Alert people. Configure job notifications for failures.
  • Watch the clock. Jobs have runtime limits and share resources with the storefront; schedule heavy work off-peak and keep transactions short.

Idempotent imports

Jobs get run twice: an operator clicks "run" again after a timeout, a scheduler retries, a file is delivered again. An import is idempotent if applying the same file twice leaves the same final state as applying it once. The general theory is in Idempotency; for feeds the practical rules are:

  1. Set state, don't add deltas. "Stock for SKU A is 12" is safe to repeat; "add 3 to stock" is not. Prefer absolute values in feeds, or carry a unique change ID.
  2. Use stable keys. Every record needs a natural ID (product ID, order number) so a repeat updates the same record instead of creating a duplicate.
  3. Skip unchanged records. Compare a hash or a version/timestamp with what you stored last time; applying nothing is the cheapest safe operation.
  4. Handle out-of-order and stale files. Keep the timestamp or sequence number of the last applied feed and ignore older files.
  5. Make progress resumable. A chunked job can restart after the last committed chunk, as long as each chunk is itself safe to repeat.
  6. Record what you did. A small status table or custom object (see Custom Objects and System Object Extensions) with the last file name and counts makes the next run decide correctly.

Live stand-ins

The first stand-in is plain JavaScript in your browser: a job runner with an error policy. Each step returns OK, WARN or ERROR, and the policy decides whether the job continues.

JS A job runner with error policies

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second stand-in runs on the Node 24 runner. It streams an XML-like inventory feed in awkwardly cut chunks (so tags are split across chunk boundaries, as network or file reads would do), parses one record at a time, and applies the changes idempotently: it keys records by SKU, skips unchanged ones using a hash, refuses a stale feed, and counts bad records against an error threshold. Running the same feed twice shows that the second run changes nothing.

Node.js Streaming a feed with an idempotent import

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The output proves the three properties that matter: records split across chunks are still parsed, a repeat of the same file changes nothing, and an older file is refused.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where do you declare a custom job step so the platform can find it?

    Custom steps are registered in steptypes.json, which names the type ID, the module path, the function and the parameters.
  2. 2. Why should a large import use a chunk-oriented step or a streaming reader?

    Reading element by element (and committing per chunk) scales to large files and allows resuming.
  3. 3. Which feed design is safest to run twice?

    Absolute values keyed by a stable ID give the same final state however many times they are applied.
  4. 4. A file contains 60% invalid records. What should the job do?

    A high share of bad records usually means a wrong or corrupted file. Failing loudly beats half-applying it.

Summary

  • A job is an ordered set of steps with a schedule, a scope (organization or site), error policies and logs.
  • Built-in steps import and export catalogs, prices, inventory and more; custom steps are declared in steptypes.json and implemented as script or chunk-oriented modules.
  • Feeds are namespaced XML; know the import mode (merge, replace, delete) and stream large files.
  • Validate early, skip and count bad records with a threshold, archive files and alert on failure.
  • Make imports idempotent: absolute values, stable keys, hashes to skip unchanged records, stale-file checks and resumable chunks.

Next: managing storefront content with Content Assets and Page Designer.