Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 42 ยท Core Data Types

Streams

Learn Redis Streams: XADD and entry IDs, XRANGE, XREAD, consumer groups with XREADGROUP and XACK, pending entries, XTRIM, and streams vs lists and pub/sub.

A Redis stream is an append-only log. Producers add entries to the end with XADD; each entry gets a unique, increasing ID and holds a few field-value pairs, like a tiny hash. Consumers read the log from any position, as many times as they like, because reading does not remove anything. On top of that log, consumer groups let a team of workers share the entries of one stream so that each entry is handled by exactly one of them, with acknowledgements and a record of what is still unfinished.

Streams were designed for event feeds, activity logs, sensor data, audit trails and work queues that must not lose messages. They fill the gap left by two older tools: a list forgets an item as soon as someone pops it, and pub/sub forgets a message the moment it has been delivered (or if nobody was listening).

Real Redis cannot run on this page, so the redis-cli snippets are read-only. The runnable examples use a plain JavaScript emulator with a fake clock, so IDs and idle times are the same on every run. It implements the log and the consumer-group bookkeeping faithfully enough to learn the semantics, but it is not Redis: it has no blocking reads, trims exactly (real ~ trimming is approximate), and keeps everything in one JavaScript array.

A log, not a queue

Append only XADD adds to the end.
Unique IDs Time plus a sequence number.
Readable again Reading never deletes entries.
1 / 2

Entries and IDs

XADD key ID field value [field value ...] appends an entry and returns its ID. Pass * as the ID and Redis generates one: <milliseconds-time>-<sequence>, for example 1700000000000-0. If two entries arrive in the same millisecond, the sequence number counts up (...-0, ...-1), so IDs always increase. You can supply your own ID, but it must be greater than the last one, otherwise Redis refuses. Because the ID starts with a timestamp, you can ask for "everything since 10:00" by using that time in milliseconds as a range bound.

Text
127.0.0.1:6379> XADD events * type signup user ana
"1700000000000-0"
127.0.0.1:6379> XADD events * type login user bo
"1700000000000-1"
127.0.0.1:6379> XLEN events
(integer) 2

Reading has two styles:

  • XRANGE key start end [COUNT n] returns entries between two IDs (inclusive). - means the smallest possible ID and + the largest. A leading ( makes a bound exclusive, which is how you paginate: ask for (last-seen-id onward. XREVRANGE goes backwards, so XREVRANGE events + - COUNT 1 is "the newest entry".
  • XREAD [COUNT n] [BLOCK ms] STREAMS key [key ...] id [id ...] returns entries with an ID greater than the one you give, for one or several streams. Remember the last ID you saw and pass it next time. The special ID $ means "only entries added after this call", which only makes sense together with BLOCK, where the server waits for new data instead of returning an empty answer.

XLEN returns the number of entries. A stream grows forever unless you trim it: XADD ... MAXLEN n or XTRIM key MAXLEN n keep roughly the newest n entries (with ~ Redis may keep a few more so it can remove whole internal blocks cheaply), and XTRIM key MINID id drops everything older than an ID. XDEL removes single entries, which leaves holes but does not change IDs.

JS A stream as an event log

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check the XRANGE events 1700000001500 1700000001500 line in the output. A bound without a sequence part means sequence 0 for the start and "the maximum" for the end, so one timestamp selects every entry created in that millisecond.

Consumer groups

With plain XREAD, every reader sees every entry. To split the work you create a consumer group: XGROUP CREATE key group id [MKSTREAM]. The id says where the group starts reading: 0 means from the beginning of the log, $ means only entries added from now on. MKSTREAM creates the stream if it does not exist yet.

Inside the group, each worker is a consumer, identified only by a name string you choose. They call:

XREADGROUP GROUP group consumer [COUNT n] [BLOCK ms] STREAMS key >

The special ID > means "entries never delivered to anyone in this group". The group remembers the last ID it handed out, so different consumers get different entries. Each delivered entry goes onto the group's pending entries list (PEL), tagged with the consumer's name, the delivery time and a delivery counter. It stays there until the consumer calls XACK key group id [id ...], which means "done, forget it". If the worker crashes before acknowledging, the entry is still pending, so nothing is lost.

Two inspection and recovery commands finish the picture:

  • XPENDING key group gives a summary (how many pending, smallest and largest ID, count per consumer). Adding start end count [consumer] lists individual entries with their owner, idle time (milliseconds since the last delivery) and delivery count.
  • XCLAIM and XAUTOCLAIM move pending entries from a silent consumer to a live one when they have been idle longer than a threshold. A high delivery count is the usual sign of a "poison" message that keeps killing workers, and the place to move it to a dead-letter stream.

Passing a concrete ID instead of > to XREADGROUP (for example 0) returns the consumer's own pending entries, which is how a restarted worker resumes what it was doing before it died.

Delivery is at least once. If a worker finishes the job but crashes before XACK, the entry will be claimed and handled again, so make your handlers idempotent (safe to run twice), for example by checking an "already processed" key first.

The next TryIt walks through a complete shipping scenario: two workers share three of four orders, one entry is acknowledged, both workers then go silent, and a third worker rescues the stuck entries after the fake clock advances a minute.

JS Consumer groups, pending entries and recovery

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the output, watch two things: XACK returns 1 the first time and 0 the second time (a double acknowledgement is harmless), and the delivery count of the two claimed entries went from 1 to 2 when w3 took them over.

Streams, lists and pub/sub

All three move messages, with different guarantees:

List Pub/sub Stream
Messages stored Until popped Not stored at all Until trimmed or deleted
Late or offline consumer Misses nothing, but items are gone once popped Misses everything sent while away Reads from any ID, including the start
Several independent readers No (each item goes to one popper) Yes, all subscribers get a copy Yes, with several readers or groups
Work sharing with acknowledgement Hand-built with LMOVE No Built in (consumer groups, PEL)
Memory Grows with the backlog Constant Grows until you trim

Use a list when a simple, short-lived queue is enough. Use pub/sub for fire-and-forget notifications where losing a message is fine (live "typing..." indicators, cache invalidation hints). Use a stream when messages matter, when several systems need the same events, or when you need to know what was delivered and what was finished. The price is bookkeeping: you must trim the stream, ack entries, and decide what to do with entries that keep failing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does XADD events * type login return?

    With * Redis generates an ID of the form milliseconds-sequence and returns it, so you can remember it as your read position.
  2. 2. A consumer reads with XREADGROUP ... STREAMS orders >. What does > ask for?

    means entries never delivered to anyone in the group. Passing a real ID such as 0 instead returns the calling consumer's own pending entries.

  3. 3. A worker crashes after receiving an entry but before calling XACK. What happens to the entry?

    Pending entries stay on the PEL until acknowledged. Another consumer can take them over with XCLAIM or XAUTOCLAIM once they have been idle long enough. Redis never moves them anywhere by itself.
  4. 4. Why should handlers for stream entries be idempotent?

    If a worker finishes the work but dies before XACK, the entry is claimed and handled a second time. Idempotent handlers make that safe.
  5. 5. Which tool forgets a message as soon as it has been delivered, even if nobody was listening?

    Pub/sub stores nothing: only the subscribers connected at that moment receive the message.

Summary

  • A stream is an append-only log; XADD returns an ID of the form milliseconds-sequence, and IDs always increase.
  • XRANGE and XREVRANGE select by ID range (with ( for exclusive bounds); XREAD returns entries after a given ID; XLEN counts.
  • Trim with MAXLEN or MINID (the ~ form is approximate) so the stream does not grow forever.
  • Consumer groups share one stream among workers: XREADGROUP ... > delivers new entries, XACK completes them, XPENDING shows what is unfinished.
  • XCLAIM and XAUTOCLAIM rescue entries from silent consumers; delivery is at least once, so handlers must be idempotent.
  • Choose streams when messages must be kept, re-read or tracked; lists suit simple queues and pub/sub suits throwaway notifications.

Next, a different kind of compact data: Bitmaps and Bitfields.