Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 70 ยท CRUD: create and read

Cursors: Iterating, Batching and Timeouts

How MongoDB cursors work: iterate with hasNext, next, forEach and toArray, tune batchSize, avoid cursor timeouts, understand exhaustion and tailable cursors, and count correctly.

When you call find(), MongoDB does not send back every matching document in one giant reply. It returns a cursor: a handle on the result set that the server keeps open while you pull documents from it a batch at a time. A cursor is why you can scan a collection with millions of documents without running out of memory, and also why some scripts fail with "cursor not found" half-way through.

This lesson explains what a cursor is, the ways to walk through one, how batching works, why cursors time out, what "exhausted" means, and how the different counting methods differ. The mongosh and driver snippets are real MongoDB syntax and cannot run in this page. The TryIts use a small stand-in that mimics a server cursor (batches, round trips, a fake clock) so you can see the behavior; it is not a real MongoDB server and does not talk to one.

A cursor in four ideas

A handle The server remembers where you are.
Batches Documents arrive a batch at a time.
Timeouts Idle cursors are closed after 10 minutes.
Exhausted Once drained, it is gone.

find() and aggregate() both return cursors.

1 / 2

What find() returns

JavaScript
const cursor = db.movies.find({ year: { $gt: 1990 } })

Nothing has been fetched yet. The cursor is a lazy description of the query. The first time you ask for a document, the driver sends the query; the server runs it, replies with the first batch and a cursor id. When you have used up the batch, the driver sends a getMore command with that id to fetch the next one. When the server has no more results it replies with cursor id 0, and the cursor is exhausted.

In mongosh, an expression that returns a cursor prints the first 20 documents and a hint. Type it to see the next 20. That display size is a shell convenience (config.set("displayBatchSize", 50) changes it), not the real batch size on the wire.

Walking a cursor

The shell and every driver offer the same few tools:

JavaScript
// hasNext() / next(): manual iteration
const cursor = db.movies.find({ year: { $gt: 1990 } }).sort({ year: 1 })
while (cursor.hasNext()) {
  printjson(cursor.next())
}

// forEach: a callback for every document
db.movies.find().forEach((doc) => print(doc.title))

// toArray(): load everything into memory (only for small results)
const all = db.movies.find({ genre: "drama" }).limit(100).toArray()

In the Node.js driver, a cursor is an async iterable, which is usually the clearest way to stream a large result:

JavaScript
const cursor = collection.find({ year: { $gt: 1990 } })
for await (const doc of cursor) {
  console.log(doc.title)
}
// or: await cursor.toArray(), await cursor.next(), await cursor.forEach(fn)

In Python's PyMongo, for doc in collection.find(...) does the same. Two rules apply in every language:

  1. Stream, do not collect, when the result is large. toArray() and list(cursor) hold every document in memory at once. Prefer a loop, add a limit, or use a projection so you pull fewer fields.
  2. A cursor is single-use. After you have read all of it, it is exhausted. To read again, run the query again.

Sorting, skip and limit are applied on the server before batching; see Sorting, Limiting and Skipping.

Batches and batchSize

The server returns documents in batches. By default the first batch holds up to 101 documents and every batch is capped at 16 MiB (the maximum BSON message size), whichever limit is reached first. Later batches are limited only by that size cap unless you set batchSize:

JavaScript
db.movies.find().batchSize(500)

batchSize controls how many documents come back per round trip. It does not change which documents match or how many you see in total; it changes how many network round trips it takes:

  • A small batch size means more getMore calls (more latency) but less memory per reply and less work wasted if you stop early.
  • A large batch size means fewer round trips but bigger replies held in memory on both sides.

limit(n) is different: it stops the query after n documents in total. If you set limit(10) and a larger batchSize, you still get 10 documents in one batch. A limit(1) or a negative limit asks for a single batch and closes the cursor right away.

The first TryIt models a server holding 250 documents. Run it, then change the batch size and the limit and compare the number of round trips.

JS Batches and round trips

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With the default batch the 250 documents need three round trips (101, 101, 48). With batchSize(1000) one trip is enough, and limit(10) ends the story after a single small batch. The real server also splits batches by size, so documents that are very large produce smaller batches than batchSize suggests.

Cursor timeouts and noCursorTimeout

A server cursor holds resources (it may hold a snapshot of the data and, if sorted, buffered results). MongoDB therefore closes a cursor that has been idle for 10 minutes (the cursorTimeoutMillis parameter). If your script reads one batch and then spends a long time processing before asking for the next, the cursor may be gone. The next getMore fails with an error like CursorNotFound: cursor id 12345 not found.

There are three honest fixes, in order of preference:

  1. Process faster or in smaller pieces. Use a smaller batchSize so each batch is processed well inside the timeout, or use a projection so batches are lighter.
  2. Page through the data yourself with a range query on an indexed field instead of one long-lived cursor: fetch 1,000 documents with _id > lastSeenId, process them, repeat. This also survives crashes, because you know where to resume.
  3. Disable the timeout with noCursorTimeout, and then close the cursor yourself:
JavaScript
const cursor = db.events.find().noCursorTimeout()
try {
  while (cursor.hasNext()) { slowProcess(cursor.next()) }
} finally {
  cursor.close()   // forgetting this leaks a server-side cursor
}

noCursorTimeout is a last resort, not a default. A cursor that never times out stays open if your program crashes without closing it. When a cursor belongs to a session (the default in modern drivers), the session's own 30-minute idle timeout still applies, so for jobs that run for hours use the paging approach.

The second TryIt shows exhaustion and the timeout with a fake clock. Notice that a cursor that is idle too long dies, and that a drained cursor cannot be read again.

JS Idle timeouts and exhaustion

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first run fails because the 15-minute pause happens between batches. The noCursorTimeout run survives it. The third run needs only one round trip, so there is nothing to time out. The last lines show that a drained cursor is finished; you cannot rewind it.

Tailable cursors

A normal cursor ends when it runs out of documents. A tailable cursor stays open after reaching the end of a capped collection (a fixed-size, insertion-ordered collection, see Capped Collections and Managing Collections) and returns new documents as they are inserted, much like tail -f on a log file. With awaitData, the server waits a short time for new data before answering an empty batch.

JavaScript
const cursor = db.log.find({}, { cursorType: "tailableAwait" })   // Node.js driver option

If the cursor reaches the end and nothing is added, it stays idle; if the document it points at is overwritten by the capped collection's wrap-around, the cursor is dead and you must reopen it. For most applications today, change streams are the better tool for reacting to new data, because they work on any collection, survive restarts with a resume token and do not require capped collections.

Counting results

Counting is a common reason to touch a cursor, and there are three different tools:

Method What it counts Notes
collection.countDocuments(filter) Exactly the documents matching filter Runs an aggregation, uses an index when it can, accurate. Use this when you need a correct number.
collection.estimatedDocumentCount() All documents, from collection metadata Very fast, takes no filter, can be slightly off after an unclean shutdown or in sharded clusters with orphaned documents.
cursor.count() / collection.count() Deprecated Replaced by the two methods above; avoid in new code.

Counting with a filter still has to examine the matching index entries or documents, so on a huge collection it can take noticeable time. If you only need to know whether any match exists, ask for one document with findOne or limit(1) instead of counting.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does find() return in the shell and in drivers?

    find() returns a cursor. It is lazy: the query runs when you first read from it, and documents arrive in batches through getMore calls.
  2. 2. What does batchSize control?

    batchSize changes the number of network round trips, not the result. Use limit() to change how many documents you get in total.
  3. 3. A long-running script fails with CursorNotFound after processing a slow batch. What is the most robust fix?

    Idle cursors are closed after 10 minutes. Paging with _id > lastSeenId lets you resume at any time. noCursorTimeout is a last resort and requires you to close the cursor.
  4. 4. Which method gives an accurate count of documents matching a filter?

    countDocuments(filter) counts exactly. estimatedDocumentCount() is fast but takes no filter and reads metadata.
  5. 5. What is a tailable cursor for?

    A tailable cursor follows the end of a capped collection like tail -f. Change streams are the more flexible modern alternative.

Summary

  • find() and aggregate() return cursors: lazy handles that fetch documents from the server in batches using getMore.
  • Walk a cursor with hasNext()/next(), forEach, a for await loop or toArray(); stream large results instead of loading them all into memory.
  • The first batch holds up to 101 documents and any batch up to 16 MiB; batchSize changes round trips, limit changes the total.
  • Idle cursors close after 10 minutes. Prefer faster processing or _id-range paging; use noCursorTimeout only with an explicit close().
  • A drained cursor is exhausted and cannot be rewound; tailable cursors follow a capped collection, and change streams are usually the better tool today.
  • Use countDocuments(filter) for an exact count and estimatedDocumentCount() for a fast total.

Next, learn how to manage collections themselves in Capped Collections and Managing Collections.