Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 42 ยท Working with Redis

Pipelining and Pub/Sub

Learn Redis pipelining to cut round trips, and Pub/Sub with PUBLISH, SUBSCRIBE and PSUBSCRIBE, including at-most-once delivery and when to use streams.

Redis can execute a command in microseconds, but your application cannot ask for it that fast. Between your code and the server sits a network, and each question-and-answer exchange across it costs a round trip. When you send 1,000 small commands one at a time, the time you wait is almost entirely network delay, not Redis work. Pipelining removes that waste by sending many commands in one go and reading all the replies together.

The second half of this lesson covers Pub/Sub, Redis's built-in broadcast mechanism: publishers send messages to named channels, and every client currently subscribed to a channel receives them. It is fast and simple, with one big caveat about delivery that you need to understand before you use it.

The code that talks to a real server is read-only here. The runnable examples are JavaScript models of the same ideas, with a fake clock so the numbers are always the same.

Why round trips dominate

Imagine a server in the same data center with a 0.5 millisecond round trip. A typical command takes a few microseconds to run, so one command costs about 0.5 ms of waiting for 0.005 ms of work. Ten thousand commands in a loop therefore take about five seconds, and the server is idle for 99% of it. Both programs are waiting: the client for the reply, the server for the next command.

One at a time

Send One command goes out.
Wait The reply crosses the network.
Repeat A full round trip for every command.

The server is mostly idle while you wait.

1 / 2

What pipelining is

Redis processes the commands on a connection strictly in order, and each reply goes back in the same order. That means a client is free to write several commands before reading any reply. The server runs them one after another and queues the replies. Client libraries expose this as a pipeline object: you add commands, then send them as a batch and receive an array of results.

JavaScript
// Node.js with ioredis (read-only: needs a server)
const pipeline = redis.pipeline();
pipeline.set("a", 1);
pipeline.incr("a");
pipeline.get("a");
const results = await pipeline.exec();
// [[null, "OK"], [null, 2], [null, "2"]]  (error and value for each command)

Three facts to remember:

  • Pipelining is not atomic. Commands from other clients can run between yours. If you need isolation, use MULTI/EXEC, which can also be sent inside a pipeline so both savings combine.
  • Errors are per command. One failing command does not stop the rest. Read every result.
  • Batches should be bounded. The server holds all replies in memory until it writes them out, so send commands in groups of a few hundred or a few thousand rather than millions at once.

Replies cannot depend on earlier replies in the same batch. If command 3 needs the answer to command 1, split them across two round trips, or move the logic into a Lua script.

Measuring the difference

The model below counts round trips. Each send call is one trip that costs a fixed latency, and each command costs a small execution time. It runs 10,000 INCR commands three ways: one at a time, in pipelined batches of 100, and in one huge batch.

JS Round trips: one by one vs pipelined

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The server work (50 ms in total) is the same in all three runs. Only the waiting changes. The numbers come from the model's fake clock and its assumed latency: on a real network the exact figures differ, but the shape is typical, and the gain grows with latency. Note also that bigger is not always better: the jump from batches of 100 to one huge batch is tiny compared with the jump from one-by-one to batches.

Many commands already accept several arguments (MSET, MGET, SADD with many members), which is even cheaper than a pipeline when it fits. Use a pipeline when the commands differ.

Pub/Sub

Pub/Sub decouples senders from receivers. A publisher sends a message to a channel name with PUBLISH; Redis forwards it to every connection that is subscribed to that channel at that moment, and replies with the number of subscribers that received it. Channels need no setup: they exist only while someone subscribes.

TEXT
# connection A
127.0.0.1:6379> SUBSCRIBE news:sports
1) "subscribe"
2) "news:sports"
3) (integer) 1

# connection B
127.0.0.1:6379> PUBLISH news:sports "Final score: 3-2"
(integer) 1

# connection A now prints
1) "message"
2) "news:sports"
3) "Final score: 3-2"

PSUBSCRIBE subscribes to a pattern, using glob style: * matches any run of characters, ? one character, [ae] a set. A client subscribed to news:* receives messages published to news:sports, news:weather and any other matching channel, as pmessage events that include both the pattern and the actual channel. UNSUBSCRIBE and PUNSUBSCRIBE undo them.

In the classic protocol (RESP2) a connection that has subscribed can issue only subscription commands, PING and a few others, so subscribers use a dedicated connection, separate from the one that runs normal commands. With RESP3 (HELLO 3) a single connection can do both, because pushed messages are a separate reply type.

At-most-once delivery

This is the caveat. Pub/Sub is fire and forget:

  • Messages are not stored. If no one is subscribed at the instant of PUBLISH, the message is gone, and the publisher sees 0.
  • A subscriber that disconnects (a crash, a restart, a network blip) misses everything sent while it was away. There is no history and no replay.
  • There are no acknowledgements. Redis does not know whether your handler ran.
  • A subscriber that reads too slowly can be disconnected by the server when its output buffer grows beyond the configured limit.

That is the right trade-off for live notifications, cache-invalidation hints, "someone is typing" signals and chat fan-out where a missed message is acceptable. It is the wrong tool for work that must not be lost.

JS A Pub/Sub broker with patterns and a late subscriber

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the three facts the output shows. PUBLISH returns the count of receivers, and 0 when nobody listens. Bob, subscribed to the pattern, receives pmessage entries for both news channels. And Carol never sees the sun message sent while she was disconnected: it is simply lost. The model reproduces Redis's delivery rules; it does not model sockets, buffers or RESP.

Sharded Pub/Sub

In a classic Redis Cluster, every PUBLISH is forwarded to every node, so traffic grows with cluster size. Redis 7 added sharded Pub/Sub: SSUBSCRIBE and SPUBLISH work like their classic versions, but the channel name is hashed to a slot (see Redis Cluster) and the message stays inside the shard that owns it, plus its replicas. Use it for high-volume channels in Cluster, and remember that patterns are not supported for sharded channels.

When to use streams instead

If messages must survive a restart, be processed once by a group of workers, or be replayed, use a stream with consumer groups, or a list-based queue. The comparison is short:

Need Pub/Sub Streams
Messages stored for later readers No Yes, until trimmed
Acknowledgements and retries No Yes (XACK, pending list)
Several workers share the load No (each subscriber gets a copy) Yes (consumer groups)
Lowest overhead, live fan-out Yes Slightly more

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does pipelining speed up a loop of 10,000 small commands?

    The time is dominated by network waits. Sending commands in batches pays the round trip once per batch instead of once per command.
  2. 2. Is a pipeline atomic?

    A pipeline only batches the network traffic. For isolation, wrap the commands in MULTI and EXEC, which can themselves be pipelined.
  3. 3. A subscriber crashes for a minute while messages are published. What happens when it reconnects?

    Pub/Sub is at-most-once with no history. Use streams when subscribers must not miss messages.
  4. 4. What does PUBLISH reply with?

    PUBLISH returns an integer count of receivers, which can be 0.
  5. 5. Which pattern matches news:sports and news:weather?

    Only PSUBSCRIBE treats * as a glob; SUBSCRIBE takes literal channel names, and sharded channels do not support patterns.

Summary

  • Round trips, not Redis, dominate the time of many small commands; pipelining sends a batch and reads the replies in order.
  • Pipelines are not atomic and report errors per command; keep batches to a few hundred or thousand commands.
  • Pub/Sub broadcasts to current subscribers only: PUBLISH replies with the receiver count, PSUBSCRIBE matches glob patterns, and subscribers normally use a dedicated connection.
  • Delivery is at-most-once: nothing is stored, acknowledged or replayed.
  • Sharded Pub/Sub (SSUBSCRIBE/SPUBLISH) keeps messages within one shard in Cluster.
  • Use streams or list queues when messages must not be lost.

Next, learn how to run logic atomically on the server in Lua Scripting and Functions.