Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 45 ยท Data and Analytics

Pub/Sub Messaging

Learn Pub/Sub on Google Cloud: topics, subscriptions, push vs pull, acknowledgements, ordering, dead-letter topics and at-least-once delivery, with a broker stand-in.

When one part of a system needs to tell another part that something happened, calling it directly is fragile. If the receiver is slow, down or busy, the sender waits or fails. Pub/Sub is Google Cloud's managed messaging service: the sender publishes a message to a topic and moves on, and any number of receivers get a copy through subscriptions, each at its own speed. The two sides never need to know about each other, and a burst of traffic is absorbed instead of crashing the receiver.

Typical uses are sending events from an app to analytics, fanning out an "order placed" event to billing, email and inventory, and feeding streaming pipelines such as Dataflow. The service is global, scales without capacity planning, and charges mainly by the volume of data published and delivered.

The Pub/Sub vocabulary

Publisher Sends messages to a topic.
Topic A named channel for messages.
Subscription Each one gets its own copy of every message.
Subscriber Receives, processes, then acknowledges.
1 / 2

Topics and subscriptions

A topic is where publishers send messages. A subscription is attached to one topic and represents one consumer application (or one group of identical workers sharing the load). Two subscriptions on the same topic each receive every message, like two newspaper subscribers. Several workers pulling from the same subscription share its messages, so each message goes to only one of them. This is how you scale a consumer.

Terminal
# Read-only: these commands need a real project.
gcloud pubsub topics create orders

gcloud pubsub subscriptions create billing-sub \
  --topic=orders --ack-deadline=30

gcloud pubsub topics publish orders --message='{"orderId": 1234}'

A message has a payload (bytes, often JSON), optional string attributes (key/value metadata useful for filtering), a message ID and a publish time. Unacknowledged messages are kept for a retention period (seven days by default, configurable), so a consumer that is down for an afternoon catches up when it returns. Subscriptions can also have a filter on attributes, so a subscriber only sees the messages it cares about.

Push or pull

You choose how a subscription delivers messages:

  • Pull: your subscriber asks for messages, processes them, and acknowledges them. You control the pace, which suits workers on Compute Engine or GKE and batch jobs.
  • Push: Pub/Sub sends each message as an HTTPS POST request to an endpoint you provide, such as a Cloud Run service. The service replies with a success status (such as 200 or 204) to acknowledge; any other response, or a timeout, means "try again later". Push needs no polling loop, which makes it ideal for serverless code that scales to zero.
  • Export subscriptions write messages straight to BigQuery or Cloud Storage with no code at all.

Secure a push endpoint: require authentication, and configure the subscription to send a signed token from a service account, so only Pub/Sub can call your service (see Service Accounts and Credentials).

Acknowledgements and the deadline

When a subscriber receives a message, the clock starts on the acknowledgement deadline (10 seconds by default; you can set it from 10 to 600 seconds, and client libraries extend it automatically while your handler is still running). If you acknowledge (ack) before it expires, the message is finished. If you send a negative ack, or the deadline passes with no reply, Pub/Sub redelivers it.

This leads to the most important property of the service: delivery is at-least-once. Every message is delivered one or more times. Duplicates can occur, for instance when a worker is slow, or crashes after doing the work but before acknowledging. (Pull subscriptions in a single region can optionally enable exactly-once delivery, which narrows the duplicates, but your code should still be written defensively.)

The consequence for you: make your handler idempotent, meaning running it twice has the same effect as running it once. Typical tricks are to store processed message IDs (or a business key such as an order ID) and skip repeats, to use "set" operations instead of "add", or to rely on unique constraints in the database.

Ordering

By default there is no guarantee about the order in which messages arrive. If order matters for related messages, such as all events for one customer, publish them with an ordering key (for example the customer ID) to a topic whose subscription has message ordering enabled. Messages with the same key are delivered in the order they were published, while different keys flow in parallel. Ordering costs some throughput and ties a key to one publishing region, so use it only where needed. If one message in a key fails and is redelivered, later messages with that key wait for it.

Dead-letter topics

Some messages can never succeed: malformed data, or a bug triggered by one special input. Without a limit, such a poison message would be redelivered forever. A dead-letter topic solves this: you set a maximum number of delivery attempts on the subscription (between 5 and 100), and after that many failures Pub/Sub forwards the message to another topic, where you can inspect it or alert on it.

Terminal
gcloud pubsub subscriptions update billing-sub \
  --dead-letter-topic=orders-dead \
  --max-delivery-attempts=5

Pub/Sub's service account needs permission to publish to the dead-letter topic and to acknowledge messages on the subscription, or forwarding silently fails to work. You can also configure retry policy with exponential backoff so a struggling endpoint is not hammered with instant retries.

Stand-in: a broker with acks and retries

The real service is a distributed system you cannot run here. The stand-in below is a tiny in-memory broker with a simulated clock. It models the rules that matter: every subscription gets a copy, delivery starts an ack deadline, a missing ack means redelivery, and after a maximum number of attempts the message moves to a dead-letter topic. Real Pub/Sub has many more details (batching, flow control, regions); the attempt count of three is only to keep the output short.

JS Stand-in: acks, retries and a dead-letter topic

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two subscriptions, two behaviors: email received all three messages untouched, while billing retried the poison message at 5-second intervals and gave up after the third attempt, so the good orders were never blocked behind it.

Stand-in: why handlers must be idempotent

In the second model a worker is slower than the ack deadline, so the message arrives a second time. The naive handler charges twice; the idempotent handler remembers the message ID.

JS Stand-in: duplicate delivery and idempotency

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In production, the "already handled" set lives in a database or cache that all workers share, not in memory.

Choose the ack deadline to match how long your handler really takes, and keep handlers short. If one message needs minutes of work, let the library extend the deadline, or hand the job to a task queue and ack quickly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Two subscriptions are attached to the same topic. A message is published. What happens?

    Every subscription on a topic receives every message. Workers sharing one subscription divide its messages instead.
  2. 2. What delivery guarantee does Pub/Sub give by default?

    A message can be delivered more than once, so handlers must tolerate duplicates.
  3. 3. A subscriber receives a message but does not ack it before the deadline. What happens?

    No ack means Pub/Sub assumes processing failed and sends the message again.
  4. 4. Which feature handles a message that fails every time?

    After the configured number of failed attempts the message is forwarded to a dead-letter topic for inspection.
  5. 5. Which is the right way to keep messages for the same customer in order?

    Messages with the same ordering key are delivered in publish order when ordering is enabled.

Summary

  • Publishers send to topics; each subscription gets a copy of every message, and workers on one subscription share them.
  • Pull gives control of pace; push calls an HTTPS endpoint such as Cloud Run; export subscriptions write to BigQuery or Cloud Storage.
  • Delivery is at-least-once: ack after success, expect duplicates, and make handlers idempotent.
  • Use ordering keys when related messages must stay in order, and dead-letter topics for poison messages.

Next: Dataflow, Dataproc and Batch Pipelines