Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 70 ยท Transactions and consistency

Change Streams and Real-Time Updates

Learn MongoDB change streams: watch() a collection, read insert, update and delete events, filter with a pipeline, resume after a crash and keep caches in sync.

Most applications eventually need to react to data changes. A new order should trigger an email, a changed price should clear a cache entry, and a deleted account should remove a search record. You could poll the collection every few seconds, but polling wastes queries and always arrives late. A change stream turns the question around: you ask MongoDB to tell you when something changes, and it pushes an event to your application as soon as the change is committed.

This lesson shows how to open a change stream with watch(), what the events look like, how to filter them, how a resume token lets a crashed consumer pick up exactly where it stopped, and where change streams fit in real systems. Change streams build on the oplog from Replica Sets and on the majority-commit idea from Read and Write Concerns.

A real change stream needs a real server, which this site does not have. The mongosh and driver listings are real commands and are read-only. The runnable examples use a small stand-in written in plain JavaScript: a collection that records every write in a log and pushes events to listeners. It copies the shape of the events and the resume behavior, but it is not MongoDB.

Push, do not poll

Events The server pushes each change as it is committed.
Filters An aggregation pipeline keeps only the events you want.
Resume Every event carries a token to continue from.
1 / 2

Opening a change stream

You call watch() on a collection, a database or a whole deployment. The shell and every driver return a stream you can iterate or attach a listener to. In mongosh it looks like this:

JavaScript
const stream = db.orders.watch()

while (!stream.isExhausted()) {
  if (stream.hasNext()) {
    printjson(stream.next())
  }
}

In the Node.js driver you would use collection.watch() and either for await (const event of stream) or stream.on("change", handler). The stream stays open, and each loop pass or callback receives one change event. Watching a database reports every collection in it, and watching the client reports every database except the internal ones. Most code watches a single collection.

A stream only reports changes that happen after it opens, unless you give it a starting point (a resume token or an operation time, covered below). It does not replay history on its own.

What a change event contains

Every event is a document. These fields appear most often:

  • _id: the resume token of this event. Store it if you want to continue later.
  • operationType: what happened, such as insert, update, replace, delete, drop, rename or invalidate.
  • ns: the database and collection, as { db, coll }.
  • documentKey: the _id of the changed document (and the shard key fields on a sharded collection).
  • fullDocument: the whole new document, present for insert and replace.
  • updateDescription: for update events, the updatedFields, the removedFields and any truncated arrays.

Notice what is missing. An update event does not contain the whole document, only the fields that changed. A delete event contains only the documentKey, because the document is gone. If you need the document after an update, ask for it when you open the stream:

JavaScript
db.orders.watch([], { fullDocument: "updateLookup" })

With updateLookup the server looks the document up when it builds the event and adds it as fullDocument. Be careful with its meaning: it is the document as it is at lookup time, which may be newer than the update that produced the event if more writes arrived in between. If the document was deleted in the meantime, fullDocument is null. When you need the document as it was before or right after one specific change, MongoDB 6.0 and later can store pre-images and post-images for a collection (changeStreamPreAndPostImages) and return them with fullDocumentBeforeChange and fullDocument: "required". That costs extra storage, so enable it only where you need it.

Filtering with a pipeline

watch() accepts an aggregation pipeline as its first argument, the same kind you will meet in The Aggregation Pipeline. Only a few stages are allowed ($match, $project, $addFields, $set, $unset, $replaceRoot, $replaceWith and $redact), but $match covers most needs:

JavaScript
db.orders.watch([
  { $match: { operationType: { $in: ["insert", "update"] } } },
  { $match: { "fullDocument.status": "paid" } }
], { fullDocument: "updateLookup" })

Filtering on the server means unwanted events never cross the network. One caveat: you must not modify the _id field of the events in $project or $replaceRoot, because _id is the resume token and the stream would fail.

The first TryIt: events and filters

The stand-in below keeps a log of changes and delivers them to every open stream. It supports insertOne, updateOne (with $set-style changes), deleteOne, watch(pipeline, options) with a simple $match, and fullDocument: "updateLookup". Three streams watch the same collection, so you can compare what each one sees. Change the operations, add a fourth stream, or filter on fullDocument.status.

JS Three streams on one collection

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the three outputs. The unfiltered stream sees all five changes. The filtered stream sees two inserts and one delete, and never the updates. The lookup stream shows the full document after each update, which the plain update event does not carry. The last event printed in full has the shape described above: a token in _id, the namespace in ns, and the documentKey.

Resume tokens

Applications crash, deploy and restart. A consumer that loses its stream must not lose changes. That is what the resume token is for. Every event's _id is a token that marks its position in the oplog. After you process an event, save its _id somewhere durable (a database, a file, a key-value store). When the consumer starts again, hand the saved token to watch():

JavaScript
db.orders.watch([], { resumeAfter: savedToken })

The server replays every change after that point and then continues live. resumeAfter continues strictly after the token's event. startAfter does the same but can also start a new stream after an invalidate event (for example after the collection was dropped), which resumeAfter cannot. For a starting point in time rather than a token, startAtOperationTime accepts a timestamp. Official drivers also resume automatically after temporary network errors and elections, using the last token they saw, so a brief failover does not end your stream.

A token is only useful while its change is still in the oplog, the capped log that replication uses. The oplog has a fixed size, and old entries are overwritten. If your consumer stays down longer than the oplog covers, the resume fails with a ChangeStreamHistoryLost error, and the consumer must rebuild its state from the collection itself and then open a fresh stream. Size the oplog for your longest realistic outage, and watch the "oplog window" metric.

Two more details matter for correctness. First, change streams deliver only changes that are majority-committed, so you never see an event for a write that is later rolled back. Second, events are ordered, but delivery is at-least-once from your point of view: if the consumer crashes after handling an event and before saving its token, it will see that event again after restart. Make handlers idempotent, so that applying the same event twice does the same thing as applying it once.

The second TryIt: a cache that survives a crash

This example builds the most common use of change streams: keeping a cache in step with the database. The consumer stores the last token it handled. It "crashes" (the stream is closed), more writes happen while it is down, and the restarted consumer resumes from the saved token and catches up. At the end, a truncated oplog shows what happens when the history is gone.

JS Cache invalidation with a resume token

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the output from top to bottom. Consumer 1 handles two updates. While it is down, a price changes, a product is deleted and another is added, so the cache is stale. Consumer 2 resumes from the saved token, replays exactly the three missed events and the cache matches the collection again. The stand-in also shows why the resume token is a position rather than a timestamp: it identifies an event, so nothing is skipped and nothing is replayed twice (as long as the token was saved after handling).

Where change streams fit

Change streams are a good fit for:

  • Cache invalidation: remove or refresh a cache entry when its document changes, as in the second TryIt.
  • Search and analytics sync: copy changes to a search index or a data warehouse without dual writes in your application code.
  • Notifications and workflows: send an email when an order becomes paid, or start a job when a document is inserted.
  • Real-time interfaces: push updates to browsers through WebSockets or server-sent events. The change stream feeds your server, and your server fans out to clients (Server-Sent Events shows the delivery half).
  • Auditing: record who changed what, when combined with pre-images.

They are not a message queue. They do not give you consumer groups, retries per message or back-pressure. If many workers must share the load, read the stream in one process and put jobs on a real queue. Also remember that each open stream holds a server cursor, so open one per consumer process instead of one per user connection.

One more rule of thumb: change streams tell you that a change happened, and the data in the event may already be old by the time you act on it. Treat the event as a hint to re-read the document when exactness matters.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a change stream need in order to work?

    Change streams are built on the oplog, which only replica sets and sharded clusters keep. A single-node replica set is enough for development.
  2. 2. A consumer saved the _id of the last event it handled and crashed. How does it continue without losing changes?

    The event _id is a resume token. Passing it as resumeAfter replays every change after that point, as long as it is still in the oplog.
  3. 3. You receive an update event and need the whole document. What do you do?

    Update events carry only updateDescription by default. updateLookup adds the current document (which may be newer than the change, or null if it was deleted).
  4. 4. Why should event handlers be idempotent?

    Delivery is effectively at-least-once. If the token is saved after the work and the process dies in between, the next run repeats the last event.

Summary

  • watch() on a collection, database or deployment opens a stream of change events pushed by the server; it needs a replica set or sharded cluster.
  • Events carry a resume token in _id, operationType, ns, documentKey and, depending on the type, fullDocument or updateDescription.
  • Updates do not include the whole document unless you ask for updateLookup (or pre/post images in 6.0+); deletes carry only the documentKey.
  • A $match pipeline filters events on the server, but never change the event _id.
  • Save the token after handling each event and reopen with resumeAfter; it works only while the oplog still covers that point.
  • Handlers must be idempotent, and change streams are not a replacement for a message queue.

Next, look at how the oplog and elections work in Replica Sets.