Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 42 ยท Admin, APIs and Backend Services

Cron Jobs and Message Queues

Schedule Magento cron jobs with crontab.xml, read the cron_schedule table, and move slow work into message queues with publishers, topics and consumers.

Some work should not happen while a customer waits for a page: sending a batch of emails, rebuilding a feed, syncing stock with a warehouse, cleaning old rows. Magento gives you two tools for this. Cron jobs run code on a schedule, such as every night at 2:00. Message queues let a request say "please do this later" by publishing a message, and a separate worker process, the consumer, picks it up and does the job.

In this lesson you will learn how crontab.xml registers a job, how the cron_schedule table tracks each run, how queue configuration files fit together, and what a good consumer looks like. The real Magento code and XML are read-only here, because Magento cannot run on this site. The runnable examples are plain PHP stand-ins: a scheduler that matches cron expressions and a queue with publishing, consuming and retries. For a look at how an event can start such work, see Events and Observers.

Cron jobs in Magento

Magento does not run its own timer. A single operating-system cron entry runs bin/magento cron:run every minute (bin/magento cron:install writes the entry). Each run does two things: it generates pending schedule rows for upcoming job runs, and it executes the rows that are due. Everything is stored in the cron_schedule table. If the system cron is not set up, nothing runs, which is the most common cause of "my job never fires".

You register a job in etc/crontab.xml (read-only):

XML
<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/crontab.xsd">
    <group id="default">
        <job name="acme_badge_cleanup" instance="Acme\Badge\Cron\Cleanup" method="execute">
            <schedule>0 2 * * *</schedule>
        </job>
    </group>
</config>

And the class (read-only):

PHP
namespace Acme\Badge\Cron;

use Psr\Log\LoggerInterface;

class Cleanup
{
    public function __construct(private LoggerInterface $logger)
    {
    }

    public function execute(): void
    {
        $this->logger->info('Acme badge cleanup started');
        // delete expired badges through a repository or resource model
    }
}

The details:

  • name must be unique across the whole installation. Use your vendor name as a prefix.
  • instance is a class created by the object manager, so constructor injection works (see The Object Manager and Constructor Injection). method is called with no arguments.
  • <schedule> uses the usual five fields: minute, hour, day of month, month, day of week. 0 2 * * * means 02:00 every day. Instead of a fixed schedule, <config_path> can name a system configuration path that holds the expression, so a merchant can change the time in the admin (see System Configuration (system.xml)).
  • A group collects jobs that run together. The default group covers most jobs; core modules also define groups such as index. Each group runs in its own process, and its timing settings (how far ahead schedules are generated, how long history is kept, and so on) come from etc/cron_groups.xml and the admin configuration under Stores > Configuration > Advanced > System > Cron. A heavy job in its own group does not hold up the others.

Jobs must be idempotent and quick. A run can be repeated after a crash, and two runs can overlap if a job takes longer than its interval. Process data in small batches and make "nothing to do" the cheap path.

The cron_schedule table

Each planned run is one row. The important columns are job_code, status, scheduled_at, executed_at, finished_at and messages. The statuses are:

status meaning
pending Scheduled for the future or due now
running Started and not finished yet
success Finished without an exception
error Threw an exception; the message is stored in messages
missed Its time passed without being started, for example because cron was not running

To debug a job, query this table: SELECT job_code, status, scheduled_at, messages FROM cron_schedule WHERE job_code = 'acme_badge_cleanup' ORDER BY scheduled_at DESC LIMIT 10. Rows stuck in running after a crash are a common sign of a killed process, and cleaning them up is a normal ops step. You can also run a job directly while testing with bin/magento cron:run --group=default.

A scheduler stand-in

The following stand-in implements the heart of the mechanism with fixed dates, so the output is always the same. It parses simple cron fields (*, a number, a list like 0,30, or a step like */15), generates the schedule rows for a time window the way the generator does, and then executes the due ones. One job throws on purpose, so you can see an error row and a missed row.

stateDiagram-v2
    [*] --> pending: schedule generated
    pending --> running: due, cron:run picks it
    running --> success: finished
    running --> error: exception
    pending --> missed: time passed, never started

A schedule row moves from pending to running to success or error; a row that is never picked up becomes missed.

PHP Generate and run a cron_schedule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Rows older than 60 minutes at the time of the run are marked missed in this stand-in, so everything before 01:30 gets that status. The 02:00 cleanup ran, and the feed export recorded its exception in messages instead of stopping the other jobs. Real Magento runs each job in a try/catch in the same spirit, writing the failure to the table and carrying on.

Message queues

A queue decouples "something happened" from "do the slow work". A request handler publishes a small message, such as "order 1005 was placed", and returns at once. A consumer later reads the message and does the heavy part: a call to an external system, an export, a large update. Magento's queue framework supports a MySQL-backed queue out of the box, so it works without extra services, and can use RabbitMQ (AMQP) for higher volume when it is configured in env.php. Your publisher and consumer code is the same for both.

A queue topic is configured with several small XML files in etc/. You will see the same names appear in each, which is how they are linked (read-only):

XML
<!-- etc/communication.xml: the topic and the type of its message -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="acme.badge.recalculate" request="Acme\Badge\Api\Data\RecalcMessageInterface"/>
</config>

<!-- etc/queue_publisher.xml: where messages for the topic are sent -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:MessageQueue/etc/publisher.xsd">
    <publisher topic="acme.badge.recalculate">
        <connection name="db" exchange="magento-db"/>
    </publisher>
</config>

<!-- etc/queue_topology.xml: the exchange and the binding to the queue -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:MessageQueue/etc/topology.xsd">
    <exchange name="magento-db" connection="db">
        <binding id="acmeBadgeRecalc" topic="acme.badge.recalculate"
                 destinationType="queue" destination="acme.badge.recalculate"/>
    </exchange>
</config>

<!-- etc/queue_consumer.xml: the worker and its handler -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:MessageQueue/etc/consumer.xsd">
    <consumer name="acme.badge.recalculate" queue="acme.badge.recalculate"
              connection="db" handler="Acme\Badge\Model\RecalcHandler::process"/>
</config>

Read it as a chain: the topic names the message type, the publisher says which exchange receives it, the topology routes the exchange to a queue, and the consumer reads that queue and calls a handler method. Publishing is a single call on Magento\Framework\MessageQueue\PublisherInterface:

PHP
$this->publisher->publish('acme.badge.recalculate', $message);

The message is a small data object. Put ids in it, not whole entities, and let the consumer load fresh data. By default, cron runs the consumers (a job called consumers_runner), starting each consumer, letting it process a limited number of messages, and then stopping so memory does not grow. Which consumers cron starts, and the limits, are set in app/etc/env.php under cron_consumers_runner. You can run one by hand while developing:

TEXT
bin/magento queue:consumers:list
bin/magento queue:consumers:start acme.badge.recalculate --max-messages=100
flowchart LR
    P["Publisher<br/>(request or observer)"] -->|"publish(topic, message)"| X["Exchange"]
    X -->|"binding"| Q["Queue"]
    Q --> W["Consumer<br/>(runs from cron)"]
    W --> H["Handler<br/>(your code)"]

A message travels from the publisher through an exchange and a binding to a queue, where a consumer hands it to your handler.

A queue stand-in

Delivery in most queue systems is at least once: a message may arrive twice, for example when a consumer crashes before it confirms the message. So a handler has to be idempotent. The stand-in below shows publishing, a consumer with a message limit, an idempotent handler, and a simple retry policy: a failing message goes back to the queue until it reaches a maximum number of attempts and is then set aside. This retry policy is our simplification; do not assume Magento's built-in behavior is identical, and check the version you run.

PHP Publish, consume and retry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Order 1001 was published twice but caused only one real effect, because the handler remembers what it has done. Order 1002 failed twice with a temporary error and succeeded on the third attempt. Order 1003 can never succeed, so it ends in the dead letters instead of blocking the queue forever. Always plan what happens to messages that keep failing.

Choosing between cron and a queue

Use cron when the trigger is the clock: nightly cleanup, hourly sync, daily report. Use a queue when the trigger is an event and there can be many of them: each order, each product save, each webhook. They combine well, as in Project: Order Events and Notifications, where an observer publishes a message and a scheduled retry deals with leftovers. For work that must happen immediately and cannot fail, do it in the request and not in a queue.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must be present on the server for any Magento cron job to run?

    Magento has no timer of its own. The system cron starts cron:run, which generates and executes the rows in cron_schedule.
  2. 2. A cron_schedule row has status missed. What does that mean?

    Missed rows usually mean cron was not running or was running too late. An exception produces the error status instead.
  3. 3. Which file lists the consumer that reads a queue and the handler method it calls?

    communication.xml defines the topic, queue_publisher.xml the publishing target, queue_topology.xml the routing, and queue_consumer.xml the consumer and its handler.
  4. 4. Why should a queue handler be idempotent?

    At-least-once delivery means duplicates can happen. An idempotent handler gives the same final result no matter how many times it sees a message.

Summary

  • Cron jobs are registered in etc/crontab.xml with a unique name, a class and method, and a five-field schedule or a config_path; one system cron entry running cron:run drives all of them.
  • cron_schedule records each planned run with a status (pending, running, success, error, missed), which is the first place to look when debugging.
  • Queues split work into publisher, exchange, queue and consumer, configured by communication.xml, queue_publisher.xml, queue_topology.xml and queue_consumer.xml.
  • Publish small messages (ids, not entities) with PublisherInterface::publish(); cron starts consumers, or you can start one by hand with queue:consumers:start.
  • Handlers must be idempotent and need a plan for messages that keep failing.
  • Use cron for time-based work and queues for event-based work.

Next: Custom CLI Commands