Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Indexes and Performance

Bulk Writes, Cursors and Connection Pools

Move lots of data efficiently with Mongoose - insertMany, bulkWrite, streaming cursors, batch processing with memory limits and connection pool sizing, with runnable stand-ins.

Single-document operations are fine for web requests. Imports, migrations, nightly jobs and reports touch thousands or millions of documents, and then the cost of network round trips and memory dominates. This lesson shows how to batch writes, stream reads, and configure the connection pool.

Why batching matters

Each await Model.create(...) is a separate request and response. At 1 ms of network latency per call, 100,000 inserts take over 100 seconds even if the database is instant. One request carrying 1,000 documents takes a few milliseconds. The stand-in below simulates a fixed latency per round trip.

Node.js One by one versus batched (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

insertMany

Model.insertMany(array) sends documents in batches, runs validation and middleware for each one, and is much faster than a loop of create().

JavaScript
const result = await Product.insertMany(rows);

By default it is ordered: the first error stops the rest (documents before it are kept, since MongoDB has no rollback here). To insert as many as possible and collect failures, pass { ordered: false }, and { rawResult: true } to see which failed. Mongoose validates everything up front; use { lean: true } to skip document hydration of the returned values for bigger imports. For truly huge loads, send chunks (1,000 to 10,000 documents) so a single request stays well below the 16 MB command limit and memory stays flat.

bulkWrite: mixed operations in one request

bulkWrite sends many different writes: inserts, updates, replaces and deletes together.

JavaScript
await Product.bulkWrite([
  { insertOne: { document: { sku: 'A-1', stock: 10 } } },
  { updateOne: { filter: { sku: 'B-2' }, update: { $inc: { stock: -1 } } } },
  { updateOne: { filter: { sku: 'C-3' }, update: { $set: { price: 9.5 } }, upsert: true } },
  { deleteMany: { filter: { discontinued: true } } },
], { ordered: false });

Two differences to remember. First, bulkWrite casts values to your schema types and applies defaults for inserts, but it does not run validators or save middleware (no pre('save')) the way insertMany and save() do. Pass { validateBeforeSave: true } on each insertOne, or validate in your own code, if rules matter. Second, the result tells you what happened: insertedCount, matchedCount, modifiedCount, upsertedCount, deletedCount.

Node.js Interpreting a bulk result (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Cursors: streaming reads

await Model.find() loads all matching documents into memory. For a million rows that crashes the process. A cursor fetches in batches and yields one document at a time:

JavaScript
const cursor = Order.find({ status: 'paid' }).lean().cursor({ batchSize: 500 });

for await (const order of cursor) {
  await exportRow(order);   // process one at a time; memory stays flat
}

Because the loop awaits each item, a slow consumer naturally slows the reading (backpressure). Compare with the async-iterator ideas in Node.js async iterators. Cursors can also be built from aggregations: Order.aggregate(pipeline).cursor({ batchSize: 500 }). A server-side cursor times out after about 10 minutes idle, so do not pause for long between items; for a very long job, page by _id ranges instead.

Node.js Constant memory with batches (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Processing in bounded parallel batches

A cursor handles one document at a time; sometimes you want a few in parallel without flooding the database. Collect a small batch from the cursor, process it with Promise.all, then continue:

JavaScript
let batch = [];
for await (const doc of cursor) {
  batch.push(doc);
  if (batch.length === 100) {
    await Promise.all(batch.map(handle));
    batch = [];
  }
}
await Promise.all(batch.map(handle)); // the remainder

The connection pool

Mongoose uses the MongoDB driver's connection pool. One mongoose.connect() per process is enough, and concurrent operations share the pool.

JavaScript
await mongoose.connect(process.env.MONGODB_URI, {
  maxPoolSize: 20,                 // default is 100
  minPoolSize: 2,
  serverSelectionTimeoutMS: 5000,  // fail fast if no server
  socketTimeoutMS: 45000,
});

Pool tuning rules: the pool should match what the database can take across all app instances (10 instances with maxPoolSize: 100 can open 1,000 connections). Too small and requests queue waiting for a connection; too large and the database wastes memory on idle connections. In serverless platforms (each invocation may open its own connection), cache the connection in a module variable so warm invocations reuse it, and keep the pool small.

flowchart LR
  A["Requests"] --> P["Pool (max 20)"] --> M["MongoDB"]
  A2["Request 21"] -. "waits for a free connection" .-> P

Operations beyond the pool size queue until a connection is free.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is insertMany much faster than a loop of create() calls?

    Fewer round trips: documents travel in large batches instead of one request each.
  2. 2. Which is true of bulkWrite compared with save()?

    bulkWrite is a thin, fast path: casting yes, document middleware and validators no.
  3. 3. What problem does a cursor solve?

    A cursor streams documents in batches so memory use stays flat.
  4. 4. Ten app instances each use maxPoolSize 100. What can happen?

    The pool limit is per process, so total connections add up across instances.

Summary

  • Batch writes to cut network round trips: insertMany for documents, bulkWrite for mixed operations.
  • bulkWrite skips validators and save middleware; check the result counters.
  • Stream large reads with cursors and for await, keeping memory flat.
  • Process in bounded batches to limit parallelism.
  • Size maxPoolSize by the total across all instances, and reuse one connection per process.

Next: Testing Mongoose code