Skip to content
Tutorials Center
My learning

Topics

Lesson 125 of 147 ยท Modern PHP and Ecosystem

Queues and Background Jobs

Learn how PHP queues move slow work out of the request - job serialization, workers, retries with backoff, failed jobs and idempotent job design.

A PHP request normally lives for a fraction of a second, and the user is waiting for it. Some work is too slow or too unreliable for that: sending email, resizing images, generating a PDF, calling a flaky partner API, rebuilding a search index. If you do it inside the request, pages feel slow and a failure at the third step ruins the whole response.

A queue fixes this. The request writes a small description of the work, a job, to a queue and returns immediately. Separate long-running PHP processes called workers take jobs off the queue and run them in the background. If a job fails, the worker can try again later. Laravel Queues, Symfony Messenger and libraries such as Enqueue are all built on this idea. In this lesson you will build a small queue to see each moving part, and then learn the design rules that make real systems safe.

You will need classes and objects, exceptions, JSON and the command-line PHP lesson, because workers are CLI scripts. Logging from workers is covered in Logging with PSR-3.

The moving parts

flowchart LR
    W["Web request"] -->|push job| Q[("Queue<br/>(Redis, database, SQS)")]
    Q -->|pop job| K1["Worker 1"]
    Q -->|pop job| K2["Worker 2"]
    K1 -->|failed after retries| F[("Failed jobs")]
    K1 --> E["Email, API, files ..."]

Producers push jobs, workers pop and run them, and jobs that keep failing end up in a failed-jobs store.

  • The producer is your web code. It only pushes.
  • The queue backend stores jobs durably: a database table, Redis lists, Amazon SQS, RabbitMQ. It must survive a restart.
  • The worker is a loop: take a job, run it, mark it done, repeat. Run several for more throughput.
  • A supervisor (systemd, Supervisor, a container orchestrator) keeps workers alive and restarts them.

Job serialization

A queue stores bytes, not objects, so a job must be turned into a string and back. The safest form is data, not objects: a job name plus a JSON payload. Avoid putting whole objects such as Eloquent models, open connections or closures in the payload. Pass an id and reload fresh data inside the job; by the time the worker runs, the record may have changed.

PHP's serialize() can store objects, but unserializing untrusted or old data is risky (see secure coding), and a payload created by yesterday's code might meet today's changed class. A plain name and array are easy to version.

PHP Jobs as JSON, a queue and a worker loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

In a real system the worker is a script started with something like php worker.php, running while (true) and sleeping (or blocking on Redis) when the queue is empty. Because it is a long-lived process, it keeps old code in memory: restart workers after every deploy. Memory also grows slowly, so most workers exit after a number of jobs or a memory limit and let the supervisor start a fresh one.

Retries and backoff

Jobs fail for temporary reasons: the mail server is busy, the network blinks. A worker should catch the exception, count the attempt, and put the job back with a delay. Waiting longer each time (exponential backoff) avoids hammering a service that is already struggling. After a maximum number of attempts the job moves to a failed jobs store where a human can inspect and retry it.

PHP Retries with backoff and a failed-jobs list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

Two details matter in practice. Add some random jitter to the delay so that a thousand failed jobs don't all retry at the same second. And only retry what can succeed later: a timeout is worth retrying, but a job with invalid data will fail every time, so send it straight to the failed list.

Idempotency: make a repeat harmless

Most queues promise at-least-once delivery. A worker can crash after it sent the email but before it told the queue "done", and the job will run again. So a job must be idempotent: running it twice has the same effect as running it once.

Common techniques:

  • Natural idempotence. "Set order 5 status to shipped" is safe to repeat. "Add 10 to the balance" is not.
  • An idempotency key. Store the key of processed work (a unique id in the job) and skip a duplicate. A unique database constraint makes this race-free.
  • Check before acting. Test the real state first: "if the receipt was already sent, return".
  • Pass the key to partner APIs. Payment providers accept an idempotency key so a repeated charge request returns the first result instead of charging again.
PHP An idempotent job using a processed-keys set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with fgets(STDIN)), it gets the next line from this box.

With a database, record the key and apply the change in the same transaction, so a crash cannot leave one without the other.

Other things to design for

  • Small payloads. Pass ids, not whole records.
  • Timeouts. A job that hangs blocks a worker. Set a maximum run time, and a visibility timeout on the queue so a stuck job reappears.
  • Priorities and separate queues. Keep slow, bulk jobs (reports) away from urgent ones (password reset emails).
  • Ordering. Queues rarely guarantee order. If order matters, put the sequence number in the job or chain the steps.
  • Observability. Log job start, success and failure with the job id, and watch the queue length: a growing queue means the workers are too slow or dead.
  • Graceful shutdown. Handle SIGTERM (see command-line PHP) so a worker finishes the current job before it exits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should a job payload usually contain an id instead of a full copy of a database record?

    Reloading by id gives the job the current data, and a small stable payload avoids problems when the class or record changes between push and run.
  2. 2. A queue guarantees at-least-once delivery. What does that mean for your job code?

    If a worker crashes before acknowledging, the queue delivers the job again. Idempotent jobs make the repeat harmless.
  3. 3. What is the main benefit of exponential backoff between retries?

    Increasing delays (with jitter) spread the retries out and ease the load on the failing dependency.
  4. 4. Why must you restart workers after deploying new code?

    Unlike web requests, a worker process does not reload files per job, so it keeps running the code it started with until restarted.

Summary

  • A queue lets a request hand slow or unreliable work to background workers and return quickly.
  • Store jobs as a name plus a small JSON payload (usually ids), not live objects.
  • Workers are long-running CLI processes kept alive by a supervisor; restart them on deploy.
  • Retry transient failures with exponential backoff and jitter, and move jobs that keep failing to a failed-jobs store.
  • Delivery is at-least-once, so design idempotent jobs with keys or state checks, ideally inside a transaction.
  • Watch queue length, set timeouts and shut workers down gracefully.

Next: Project: To-Do List