Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 70 ยท Projects

Project: Event Logging with Bucketing and TTL

Store a stream of sensor events efficiently in MongoDB with the bucket pattern, roll them up into daily summaries, expire old data with TTL, and meet time-series collections.

Some data never stops arriving: sensor readings, page views, audit entries, application logs. Each record is tiny, there are millions of them, and nobody cares about last year's records one by one. Storing such a stream with one document per event works at first, then your collection and indexes balloon. This project shows the standard remedies: group events into buckets, roll up old detail into summaries, and let MongoDB expire data automatically with a TTL index.

We model temperature readings from devices. The code runs on the tutorial's practice engine, a JavaScript stand-in (see The Practice Engine Used in This Tutorial): the update operators and pipelines are real MongoDB syntax, but the engine has no indexes and no background threads, so TTL expiry is shown as the effect it has, not as the mechanism. The MongoDB-native alternative, time-series collections, is real server only and is shown as read-only code.

A stream needs three tools

Buckets Many readings in one document.
Rollups Keep summaries, drop detail.
TTL Old documents delete themselves.
1 / 2

The bucket pattern

A bucket is one document that holds all readings of one device during one time window, here one hour. Instead of 4 documents for 4 readings you store 1 document containing an array of 4 readings plus a few precomputed values: a count, a running sum, a minimum and a maximum. A query for "the average temperature of device d1 at 10:00" no longer reads any array; it divides sum by count.

Writing into a bucket is a single upsert that combines several update operators, and it is atomic because it targets one document:

JavaScript
db.buckets.updateOne(
  { deviceId: "d1", hour: ISODate("2026-03-01T00:00:00Z"), count: { $lt: 4 } },
  {
    $push: { readings: { ts: ISODate("2026-03-01T00:05:00Z"), value: 21 } },
    $inc: { count: 1, sum: 21 },
    $min: { min: 21 },
    $max: { max: 21 }
  },
  { upsert: true }
)

Read the filter carefully. It names the device and the hour, so the right bucket is found. The count: { $lt: 4 } clause is the size guard: once a bucket is full, the filter no longer matches, the upsert inserts a new bucket for the same device and hour, and the full one is left alone. Without a guard a busy device would keep growing a single document towards the 16 MB limit. $min and $max set the field when it is missing, so the first reading initializes them. The next TryIt ingests twelve readings per device, shows how few documents result, and then adds an extra reading to trigger a second bucket.

JS Ingest readings into hourly buckets

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Twenty-four events became six documents, a four-to-one saving, because we chose tiny buckets of four. With real buckets of a few hundred readings you cut the document count, and the index entries pointing at them, by two orders of magnitude. The fifth reading in hour 00 opens a second bucket for the same hour, which is why queries on buckets must never assume "one bucket per hour".

Bucketing trades flexibility for efficiency. Reading one single event now means finding its bucket and filtering the array, and unbounded updates to a bucket (editing a reading in the middle) are awkward. Use it for append-only streams where you mostly ask aggregate questions.

Rollups: keep the summary, drop the detail

Raw readings are valuable for a week and almost worthless after a year, but the daily average is useful forever. A rollup is a second, much smaller collection built from the first: one document per device per day with the count, sum, minimum and maximum. Because those values are mergeable (you can add two sums and two counts), a rollup can be refreshed incrementally. An average is not mergeable by itself, so you store sum and count and divide when you read.

In the stand-in below, a pipeline groups the buckets by device and day, then an upsert per result $incs the numbers into the daily collection. Real MongoDB can do the write step inside the pipeline with $merge, which is read-only here.

TTL: let the database delete old data

A TTL index is an ordinary index on a date field with an expireAfterSeconds option. A background task on the server removes documents whose date is older than the limit. You create it once:

JavaScript
// delete each bucket 7 days after its "expireAt" date
db.buckets.createIndex({ expireAt: 1 }, { expireAfterSeconds: 0 })
// or: delete documents 30 days after "createdAt"
db.logs.createIndex({ createdAt: 1 }, { expireAfterSeconds: 2592000 })

Three facts to remember. Deletion is not instant: the background task runs about once a minute and large deletes may lag, so never rely on a TTL for precise cutoffs (filter by date in your query too). The indexed field must hold a date (or an array of dates); documents without a valid date never expire. And a TTL index is single-field only. Storing an explicit expireAt date, set when the document is written, lets you choose each document's lifetime individually, and with buckets it expires a whole hour of data in one cheap delete.

The practice engine has no background thread, so the TryIt uses a sweep(now) function that does what the TTL task does: delete every document whose expireAt is earlier than now. It runs a rollup first, then the sweep, to show the correct order of operations: summarize before the data disappears.

JS Rollup to daily summaries, then expire raw buckets

Press Run (or Ctrl + Enter) to see the console output here.

No output.

After the sweep, the two oldest buckets (hour 00 of each device, which expired at 02:00) are gone, the other four remain, and the two daily summaries still describe the full day. Run the rollup on a schedule that is shorter than the retention window, or the detail will vanish before it has been summarized.

Time-series collections

Hand-built buckets are a pattern you now understand, but MongoDB 5.0 and later can do the bucketing for you. A time-series collection is created with a timeField and, optionally, a metaField that identifies the source (the device) and a granularity hint. You insert ordinary, un-bucketed documents; the server stores them in an internal compressed bucketed format and lets you query them as if they were individual events:

JavaScript
db.createCollection("readings", {
  timeseries: { timeField: "ts", metaField: "deviceId", granularity: "minutes" },
  expireAfterSeconds: 60 * 60 * 24 * 30   // optional built-in TTL: 30 days
})

db.readings.insertOne({ ts: new Date(), deviceId: "d1", value: 21 })
db.readings.aggregate([
  { $match: { deviceId: "d1" } },
  { $group: { _id: { $dateTrunc: { date: "$ts", unit: "hour" } }, avg: { $avg: "$value" } } }
])

This code is real MongoDB only; the practice engine cannot run it. Time-series collections give you compression, automatic indexing on time and metadata, and the optional TTL shown above, in exchange for restrictions: updates and deletes of individual measurements are limited (the allowed operations depend on the server version, so check the documentation for yours), and the schema is organized around time. Choose them for sensor and metric data. Choose hand-made buckets when you need full control over the document shape, extra computed fields, or a version of MongoDB that predates them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the count: { $lt: 4 } part of the bucket filter do?

    The guard keeps buckets from growing without a limit. When a bucket is full the filter no longer matches and the upsert inserts a fresh bucket.
  2. 2. Why does the bucket store sum and count rather than the average?

    To combine buckets you add sums and counts, then divide once. Averaging averages gives wrong results when counts differ.
  3. 3. What is true about a TTL index?

    The TTL monitor runs periodically, so expiry can lag. The indexed field must contain a date, and documents without one never expire.
  4. 4. Which order is correct for a rollup and a TTL-based expiry of raw data?

    Once raw data is deleted it can no longer be summarized. The rollup must run more often than the retention window.

Summary

  • The bucket pattern stores many events in one document with precomputed count, sum, min and max, cutting document and index counts.
  • One upsert with $push, $inc, $min and $max updates a bucket atomically; a size guard in the filter starts a new bucket when one is full.
  • Rollups keep mergeable summaries (sum and count, not averages) so detail can be discarded.
  • A TTL index (expireAfterSeconds) removes expired documents in the background; it is approximate and needs a date field.
  • Time-series collections (read-only here) automate bucketing, compression and optional expiry, with restrictions on updates and deletes.

Next, protect numbers that many users change at once in Project: Inventory with Safe Concurrent Updates.