Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 70 ยท Indexes and performance

Common Performance Pitfalls

Avoid the MongoDB habits that make queries slow: unbounded arrays, large skips, unanchored regex, negations, too many indexes, small RAM and fat results.

Most slow MongoDB applications are not slow because MongoDB is slow. They are slow because of a handful of patterns that look innocent in development, when the collection has a hundred documents, and fall over in production, when it has a hundred million. The patterns repeat across projects, so you can learn to spot them in advance.

This lesson walks through the most common ones: arrays that grow forever, big skip values, regular expressions that cannot use an index, negation operators, too many indexes, a working set bigger than memory, and returning more data than you need. For each you will see why it hurts and what to do instead. Three small stand-ins let you measure the effect. They count how many index entries or bytes are involved, which is the quantity that drives real cost.

The mongosh snippets are real MongoDB commands, shown read-only. The TryIts count work in plain JavaScript (index entries visited, bytes of a document), so the ratios are meaningful but the absolute numbers are not benchmarks.

Pitfalls that read too much

Large skip Walks every entry it skips.
Loose regex Cannot narrow the index scan.
Negations Match almost everything.
Fat results Whole documents when you need two fields.
1 / 2

Unbounded arrays

Embedding comments inside a post is natural, until a post gets 50,000 comments:

JavaScript
db.posts.updateOne({ _id: 1 }, { $push: { comments: { user: "mia", text: "Nice!" } } })

An array without an upper limit has three problems:

  • The document keeps growing towards the 16 MB limit, at which point writes fail.
  • Every update to a large document means rewriting and re-indexing more data, and reading the post means loading all of it, even if the page shows ten comments.
  • If the array is indexed, the multikey index gets one entry per element, so the index grows with the array.

The fix is to bound the array or move it out: keep only the latest few items in the parent (the subset pattern), or store items in their own collection with a reference to the parent, or group them in fixed-size buckets (Schema Design Patterns, Embedding vs Referencing). If an array has a natural cap, such as "at most five phone numbers", embedding is fine.

The first TryIt measures how fast a "harmless" array grows.

JS How big does an unbounded array get?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

JSON length is only an approximation of the size BSON uses, but it is close enough to show the trend: a document under 1 KB at ten comments is over a megabyte at twenty thousand, and nothing stops it from continuing.

Large skips

Pagination with skip looks simple:

JavaScript
db.products.find().sort({ createdAt: -1 }).skip(90000).limit(10)

But skip(n) does not jump to the n-th document. The server walks and discards the first n results, so page 9,000 costs almost 9,000 times as much as page 1. The fix is range-based pagination: remember the last value you returned and ask for what comes after it (Sorting, Limiting and Skipping).

JavaScript
db.products.find({ createdAt: { $lt: lastSeenCreatedAt } }).sort({ createdAt: -1 }).limit(10)

With an index on createdAt, every page costs about the same, however deep. The trade-off is that you can go to "next" and "previous" but cannot jump straight to page 900.

Regular expressions, negations and index bounds

An index can narrow a scan to a range of entries. Some predicates produce a tight range, and others cannot:

  • An anchored, case-sensitive regex such as /^al/ becomes the range from "al" up to the next prefix, so the index scan is tight.
  • An unanchored regex such as /an/ or a case-insensitive one such as /^al/i cannot form a tight range. MongoDB can still read the index instead of the documents, but it checks every entry in it.
  • A negation ($ne, $nin, $not) matches everything except a few values, so its range is "nearly the whole index". Whether that is cheap depends on how many entries remain: $ne: "done" is fine if most documents are done, and costly if almost none are.

For text-like searching, prefer an anchored regex, a text index or Atlas Search (Text Matching and Regular Expressions). To search case-insensitively, store a normalised (lowercase) copy of the field and index that, or use a case-insensitive collation on the index.

The second stand-in counts how many entries each predicate has to examine.

JS How many index entries does each predicate read?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The anchored prefix reads only the one key that matches, while the unanchored and case-insensitive patterns read all 34 to find three and one matches. For the negations, the number of entries read is whatever is left after removing the excluded value, so status != "blocked" reads 900 of 1,000 entries to find results that an equality on the values you want would have found directly.

Too many indexes

Each index speeds up some reads and slows down every write: an insert has to add an entry to every index, an update to an indexed field has to move entries, and every index occupies disk and, more importantly, memory. A collection with fifteen indexes pays fifteen times for each insert. Habits that help:

  • Delete indexes that are prefixes of other compound indexes.
  • Use $indexStats (explain() and the Profiler) to find indexes that are never used, hide them first (Index Types), then drop them.
  • Do not index low-selectivity fields on their own, such as a boolean flag on its own, unless a partial index covers the rare value.

Working set versus RAM

MongoDB's storage engine keeps recently used data and all the index pages it needs in a cache. The working set is the data and indexes your application touches regularly. While it fits in memory, queries are served from RAM. When it does not, the same query needs disk reads, and latency jumps, often suddenly, as the data grows.

By default the storage engine cache takes about half of the memory left after reserving 1 GB (with a floor of 256 MB), and the operating system's file cache helps with the rest. Two ways to keep the working set smaller are to keep indexes lean (fewer, narrower, partial) and to keep documents small (do not store large blobs next to hot fields). Monitor cache usage and page faults in your metrics, and plan RAM for the indexes first.

Return less data

Two cheap habits remove a lot of waste:

JavaScript
db.products.find({ category: "shoes" }, { name: 1, price: 1, _id: 0 }).limit(20)
  • Use a projection (Projection) so you send only the fields you need. A product page might need the description, but the listing page does not. With a matching compound index the query can even be covered.
  • Always use limit unless you really want everything, so one careless call cannot pull a million documents across the network.

The last stand-in compares what a listing page transfers with and without projection and pagination.

JS Whole documents versus projected pages

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A quick checklist

  • Every array has a realistic upper bound, or lives in its own collection.
  • Pagination uses a range on an indexed field, not a deep skip.
  • Searches use anchored case-sensitive patterns, text search or a dedicated search engine.
  • Negation queries are checked with explain().
  • Each index has a query that needs it, and unused ones are removed.
  • Indexes (plus hot data) fit in memory.
  • Queries use projections and limits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is skip(90000) slow even with an index on the sort field?

    skip does not jump to a position. It reads past the entries it skips, so deep pages cost far more than early ones. Range-based pagination avoids that.
  2. 2. Which regular expression can use an index to narrow its scan to a tight range?

    Only an anchored, case-sensitive prefix pattern turns into a range of index keys. Unanchored and case-insensitive patterns have to test every key.
  3. 3. What is the main risk of an array that can grow without limit inside a document?

    Unbounded arrays make documents larger, writes slower and multikey indexes bigger, and a document cannot exceed 16 MB.
  4. 4. Which habit reduces both network traffic and memory use for a listing page?

    A projection returns only needed fields and a limit caps the number of documents, so less data is read, sent and held in memory.

Summary

  • Keep arrays bounded or move them to their own collection or into buckets; document size and multikey indexes grow with them.
  • Replace deep skip with range-based pagination on an indexed field.
  • Only anchored, case-sensitive regex gives a tight index range; negations read nearly the whole index when most values remain.
  • Every index slows every write and uses memory: remove prefixes and unused indexes.
  • Keep the working set (hot data plus indexes) within RAM.
  • Use projections and limits to read, send and hold less data.

Next, move on from speed to structure with Embedding vs Referencing.