Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 62 ยท Projects

Project: Async Report Generator

Build an asynchronous report generator in Symfony with a console command, a Messenger message and handler, retries, a failure transport and a mailer, with runnable stand-ins.

Some work does not belong in a web request: building a large report, resizing images, calling a slow partner API. If the user has to wait for it, the page times out; if it fails halfway, the user sees an error and has no way to retry. The standard answer is to accept the request quickly, do the work in the background and tell the user when it is done.

In this project you build such a pipeline with Symfony: a console command that requests a report, a Messenger message routed to an asynchronous transport, a handler that generates the CSV and emails it, retries with back-off, and a failure transport for messages that give up. Symfony needs a server and a worker process, so the real code is read-only. Two runnable plain-PHP stand-ins let you watch the retry logic and the full flow, including the property that makes background jobs safe: idempotency. Prerequisites: Messenger: Messages and Handlers, Console Commands and Mailer and Notifier.

The design

flowchart LR
    U["Console command<br/>or controller"] -->|"dispatch(ReportRequested)"| B["Message bus"]
    B -->|"routed"| Q[("async transport<br/>(Doctrine, Redis, AMQP)")]
    Q --> W["messenger:consume<br/>(worker)"]
    W --> H["ReportRequestedHandler"]
    H -->|"success"| M["Mailer: email with CSV"]
    H -->|"exception"| R{"Retries left?"}
    R -->|"yes"| Q
    R -->|"no"| F[("failed transport")]

The dispatcher returns as soon as the message is stored; the worker does the slow work later and decides between success, retry and failure.

Three design rules shape everything below:

  1. Messages carry identifiers, not objects. The message holds a report id and a type. The handler loads fresh data, so a message that waits an hour never acts on stale state.
  2. Handlers must be idempotent. A queue guarantees at least once delivery: a worker can crash after sending the email but before acknowledging the message, and the message is delivered again. Running a handler twice must not send two emails.
  3. Distinguish recoverable from unrecoverable errors. A timeout is worth retrying; an unknown report type never will succeed.

The message and the handler

PHP
<?php
namespace App\Message;

final readonly class ReportRequested
{
    public function __construct(
        public int $reportId,
    ) {
    }
}

The handler loads the report, exits early if the work is already done, builds the CSV and emails it. Throwing UnrecoverableMessageHandlingException skips the retries and sends the message straight to the failure transport:

PHP
<?php
namespace App\MessageHandler;

use App\Message\ReportRequested;
use App\Repository\ReportRepository;
use App\Report\CsvBuilder;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
use Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException;
use Symfony\Component\Mailer\MailerInterface;
use Symfony\Component\Mime\Email;

#[AsMessageHandler]
final class ReportRequestedHandler
{
    public function __construct(
        private ReportRepository $reports,
        private CsvBuilder $csv,
        private MailerInterface $mailer,
        private EntityManagerInterface $em,
    ) {
    }

    public function __invoke(ReportRequested $message): void
    {
        $report = $this->reports->find($message->reportId)
            ?? throw new UnrecoverableMessageHandlingException('Report no longer exists.');

        if ($report->isSent()) {
            return; // already delivered: a repeated message does nothing
        }

        $csv = $this->csv->build($report->getType());

        $email = (new Email())
            ->to($report->getRecipient())
            ->subject(sprintf('Your %s report', $report->getType()))
            ->text('The report is attached.')
            ->attach($csv, $report->getType().'.csv', 'text/csv');
        $this->mailer->send($email);

        $report->markSent();
        $this->em->flush();
    }
}

Note the order: send first, mark afterwards. A crash between the two lines means a duplicate email, which is the lesser evil compared with marking first and never sending. For stronger guarantees use a unique key on the mail provider's side or a per-report lock (Scheduler, Lock and Cron Jobs).

Transports, routing and retries

The configuration decides where the message goes and what happens after a failure:

YAML
# config/packages/messenger.yaml
framework:
    messenger:
        failure_transport: failed

        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                retry_strategy:
                    max_retries: 3
                    delay: 1000        # milliseconds before the first retry
                    multiplier: 2      # then 2000, 4000
                    max_delay: 0
            failed: 'doctrine://default?queue_name=failed'

        routing:
            App\Message\ReportRequested: async
Terminal
# development: process messages until you stop it
php bin/console messenger:consume async -vv

# production: stop after an hour or 100 messages and let the supervisor restart it
php bin/console messenger:consume async --time-limit=3600 --limit=100

# inspect and retry what failed
php bin/console messenger:failed:show
php bin/console messenger:failed:retry

In production a process supervisor (systemd or Supervisor) keeps the workers running, and workers restart after each deploy (messenger:stop-workers) so they load the new code. The stand-in below simulates exactly the retry policy above with a virtual clock, so there is no waiting: each report type fails in a different way.

PHP Retries, back-off and a failure transport (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Follow report 2: it fails twice, each retry waits twice as long as the one before, and the third attempt succeeds. Report 3 uses all three retries and then lands in the failure transport, where an operator can inspect it and replay it after fixing the cause. Report 4 never gets a retry, because retrying cannot help.

The console command

A command is the natural entry point for batch work, and it is easy to run from cron or a deploy script (Console Commands):

PHP
<?php
namespace App\Command;

use App\Entity\Report;
use App\Message\ReportRequested;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Style\SymfonyStyle;
use Symfony\Component\Console\Attribute\Argument;
use Symfony\Component\Console\Attribute\Option;
use Symfony\Component\Messenger\MessageBusInterface;

#[AsCommand(name: 'app:report:request', description: 'Queue a report and email it when ready')]
final class RequestReportCommand
{
    public function __construct(
        private EntityManagerInterface $em,
        private MessageBusInterface $bus,
    ) {
    }

    public function __invoke(
        SymfonyStyle $io,
        #[Argument('Report type: sales or stock')] string $type,
        #[Option('Recipient address')] string $email = '[email protected]',
    ): int {
        $report = new Report($type, $email);
        $this->em->persist($report);
        $this->em->flush(); // the id exists only after this line

        $this->bus->dispatch(new ReportRequested($report->getId()));
        $io->success(sprintf('Report #%d queued for %s.', $report->getId(), $email));

        return Command::SUCCESS;
    }
}

Persist before dispatching: if the message were dispatched first, a fast worker could look for a report that is not committed yet. In Symfony 7.3 and newer the #[Argument] and #[Option] attributes on __invoke() replace the older configure() and execute() pair; on earlier 7.x versions extend Command and use the classic style from the console lesson.

Putting it together

The second stand-in runs the whole flow in one script: a command parses its arguments and stores a report in SQLite, the bus queues the message, the worker loop handles it, a fake mailer records emails, and the handler is idempotent. To prove that, the script delivers one message twice, as a real queue may do after a crash.

PHP Command, bus, worker and idempotent handler (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

The second delivery of report 1 is skipped because the handler checks the stored status before doing anything, so the user gets exactly one email. The unknown report type is rejected by the command itself, before anything is queued: validate early, at the edge, so the queue only holds work that can succeed.

Operating it

  • Monitoring: alert when the failed transport is not empty. messenger:failed:show tells you the exception for each message.
  • Poison messages: a message that always fails ends up in the failure transport after its retries. Fix the code, deploy, then messenger:failed:retry.
  • Backlog: messenger:stats shows how many messages wait in each transport; add workers when the number keeps growing.
  • Testing: use the in-memory:// transport in tests and assert on $this->transport('async')->getSent(), and test the handler directly with a fake mailer (Testing with PHPUnit).
  • Deploys: workers hold the old code in memory. Restart them with messenger:stop-workers as the last deploy step (Performance and Deployment).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should a message contain a report id instead of the whole Report object?

    A message can wait in the queue for a long time. Loading the entity when the handler runs avoids acting on stale data and avoids serializing large object graphs.
  2. 2. A worker crashes after sending the email but before acknowledging the message. What will happen?

    Queues guarantee at-least-once delivery. An unacknowledged message is redelivered, which is why handlers check whether the work was already done.
  3. 3. With max_retries 3, delay 1000 and multiplier 2, how long are the waits before retries 1, 2 and 3?

    The delay starts at 1000 ms and is multiplied by 2 for each further retry.
  4. 4. When should a handler throw UnrecoverableMessageHandlingException?

    It skips the retry strategy and moves the message to the failure transport immediately, which saves pointless attempts.

Summary

  • Accept quickly, work in the background: dispatch a message, return, let messenger:consume do the slow part.
  • Messages carry ids; handlers load fresh data and are idempotent because delivery is at least once.
  • Configure retry_strategy per transport and a failure_transport for messages that give up; use unrecoverable exceptions for hopeless cases.
  • Persist before you dispatch, validate at the edge and monitor the failure transport.
  • Keep workers running under a supervisor, limit their lifetime and restart them on every deploy.

Reinforce all of this with the Practice Exercises.