Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 65 ยท Case Studies: Classic Designs

Design a Notification System

Design a notification service: channels, templates, user preferences, queues, retries with backoff, deduplication, rate limits and delivery tracking, with Python.

Notifications tell users something happened: a delivery is nearby, a friend commented, a password reset was requested. They go out as mobile push, SMS, email and in-app messages. A notification system is the shared service every other team calls instead of integrating with Apple, Google, an email provider and an SMS gateway on their own.

The tricky parts are not sending one message. They are bursts (a sale starting at noon), retries that must not duplicate, respecting user preferences, and never losing the important ones.

Requirements

  • Channels: iOS and Android push, SMS, email, in-app.
  • Types: transactional (receipts, security codes), alerts, and marketing.
  • Soft real-time: security codes within seconds; marketing can take minutes.
  • Users can opt out per channel and per type; quiet hours must be respected.
  • At least once delivery, with deduplication so users do not get the same message repeatedly.
  • Scale: 10 million notifications per day on average, peaks of 1 million in a few minutes.

Architecture

flowchart LR
    S1["Order service"] --> API["Notification API"]
    S2["Marketing tool"] --> API
    API --> V["Validate, dedupe, preferences"]
    V --> Q1["Queue: push"]
    V --> Q2["Queue: SMS"]
    V --> Q3["Queue: email"]
    Q1 --> W1["Push workers"] --> APNS["APNs / FCM"]
    Q2 --> W2["SMS workers"] --> SMSG["SMS gateway"]
    Q3 --> W3["Email workers"] --> MAIL["Email provider"]
    W1 --> L[("Delivery log")]
    W2 --> L
    W3 --> L

One queue per channel isolates failures: a slow email provider never delays push messages.

The steps:

  1. API accepts {user_id, type, template, data, idempotency_key}.
  2. Validation checks the schema and the caller's quota.
  3. Preference check loads the user's settings (opted out? quiet hours? preferred channel?) and decides channels.
  4. Template rendering merges data into per-language templates.
  5. Enqueue onto a queue per channel, with a priority (security codes before marketing).
  6. Workers call the third-party provider, with retries.
  7. Delivery log records sent, failed, delivered, opened, for debugging and analytics.

Deduplication and idempotency

The caller may retry an API call after a timeout; workers may retry after a crash. To avoid duplicates, each notification gets an idempotency key (for example order-123-shipped). The system remembers keys for a window (24 hours) and drops repeats.

Python Preferences, quiet hours and deduplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The deferred marketing message used up its key, so the retry was dropped. A real system would schedule the deferred message for the end of the quiet hours instead of relying on the caller to ask again.

Retries, backoff and dead letters

Providers fail: APNs rate-limits, an SMS gateway times out. Workers should retry with exponential backoff and jitter (wait 1 s, 2 s, 4 s, plus randomness) up to a limit, then move the message to a dead-letter queue for inspection. Permanent failures (an invalid device token, a number that does not exist) must not be retried; they should remove the token or mark the address bad.

Python Retry with exponential backoff against a flaky provider

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Jitter matters: without it, a thousand failed messages retry at the same instant and hammer the provider again.

Handling bursts and priorities

A flash sale can enqueue millions of marketing messages in minutes. Protect the important traffic:

  • Separate queues (and worker pools) per priority: security and transactional have their own, so marketing backlog never delays a login code.
  • Rate-limit per provider to stay inside their quotas, and per user to avoid spamming (for example at most 3 marketing pushes per day).
  • Batch and throttle campaigns: send at a steady speed rather than all at once.
  • Autoscale workers on queue depth.

Device tokens and tracking

Store each user's device tokens (many per user) and update them on app launch; remove tokens when a provider reports them invalid. For tracking, the delivery log captures each state change, and click and open events come back through tracking links or pixels. Keep the log in a write-optimized store, partitioned by time.

Also decide: do you need exactly once? Providers rarely give it, so the practical goal is at least once plus idempotency keys plus user-visible deduplication.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use a separate queue for each channel and priority?

    Isolation keeps a failure or burst in one lane from blocking the others.
  2. 2. What is the purpose of jitter in a retry strategy?

    Random delay breaks synchronization among many failing clients.
  3. 3. An invalid device token error comes back. What should the worker do?

    Permanent failures will never succeed. Clean up the token so future sends skip it.
  4. 4. What prevents duplicate notifications when a caller retries?

    The system drops a request whose key it has already processed.

Summary

  • A notification service centralizes channels, templates, preferences and delivery tracking.
  • Use one queue per channel and priority, with workers calling third-party providers.
  • Deduplicate with idempotency keys; retry with exponential backoff and jitter; dead-letter what fails.
  • Respect opt-outs, quiet hours and per-user rate limits; protect urgent traffic from bursts.
  • Aim for at least once delivery and log every state change.

Next: Design Search Autocomplete