Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 70 ยท Aggregation pipeline

The Aggregation Pipeline

Learn how the MongoDB aggregation pipeline passes documents through stages such as $match, $project, $sort and $limit, and why stage order matters.

find() returns documents as they are stored. But real questions are rarely that simple: "What are the five most expensive in-stock books?", "How many orders did each customer place last month?", "What is the average rating per category?" To answer them, MongoDB offers the aggregation pipeline.

A pipeline is a list of stages. Documents flow into the first stage, each stage transforms them (filter, reshape, sort, group, join) and hands the result to the next stage, and whatever comes out of the last stage is your answer. If you have used the pipe | in a shell, you already know the idea. In this lesson you will learn the stages that behave like find: $match, $project, $sort, $skip, $limit and $count. Later lessons add grouping and joins.

MongoDB cannot run on this site. The mongosh blocks are real commands and read-only. The runnable examples use a stand-in pipeline engine written in plain JavaScript over an array of documents, so the stages really execute, but it supports only the stages and operators shown here.

A pipeline is a data flow

Input All documents of a collection.
Stages Each one transforms the stream.
Output What the last stage emits.
1 / 2

Your first pipeline

You run a pipeline with aggregate, passing an array of stages. Each stage is an object with one key, the stage name starting with $:

JavaScript
db.books.aggregate([
  { $match: { stock: { $gt: 0 } } },
  { $sort: { price: -1 } },
  { $limit: 3 },
  { $project: { _id: 0, title: 1, price: 1 } }
])

Read it from top to bottom, like a recipe: keep books that are in stock, sort them by price, highest first, keep the first three, and show only title and price. The same query as a find would be db.books.find({stock: {$gt: 0}}, {title: 1, price: 1, _id: 0}).sort({price: -1}).limit(3). For simple cases find is shorter, but a pipeline can also do things find cannot, which is why it is the main tool for analysis.

aggregate returns a cursor, just like find, so you iterate it the same way.

The core stages

$match

$match keeps only documents that satisfy a filter. It uses the same query language as find: equality, $gt, $in, $or and so on (see Comparison Operators and Logical Operators). One difference: inside $match you cannot use the aggregation-expression syntax ($expr lets you).

$project

$project chooses which fields continue, and can compute new ones. It follows the rules you learned in Projection: Choosing Fields:

  • field: 1 includes a field; field: 0 excludes it.
  • _id is included unless you write _id: 0.
  • Do not mix inclusion and exclusion (except _id: 0).
  • In an aggregation $project you can also write a field path such as author: "$writer.name" to copy or rename a value, or an expression (the next lessons show many).

Once a field is projected away, later stages cannot see it. This is a classic bug: projecting out price and then sorting by price.

$sort, $skip and $limit

$sort: { price: -1 } sorts descending (1 is ascending). $skip: n drops the first n documents and $limit: n keeps at most n. For paging, sort first, then skip, then limit, and give the sort a unique tie-breaker such as _id, as discussed in Sorting, Limiting and Skipping.

$count

$count: "name" replaces the stream with one document holding the number of documents it received: { $count: "inStock" } outputs { inStock: 4 }. (If the stream is empty, the stage emits nothing at all, not a zero.)

JS A pipeline from scratch

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Stage order matters

The stages run in the order you write them, so the order changes both what you get and how much work it takes.

It changes the result. Compare "keep the 2 cheapest books, then keep those in stock" with "keep books in stock, then take the 2 cheapest". The first can return fewer than two documents, because $limit ran before $match.

It changes the cost. Put $match as early as you can: fewer documents flow through the later stages. As the first stage, $match can also use an index, just like find. Stages that must look at every document, such as $sort without an index, are cheaper when they have less to sort. A $sort immediately followed by a $limit is even better: the server only has to remember the top N documents instead of sorting everything.

A good default order is: $match (shrink), then $sort and $limit if you need only the top few, and $project last to shape the output (the optimizer can also move some of this for you, as you will see in Aggregation Performance and Explain). Reorder only when the logic requires it.

flowchart LR
    A["5 documents<br/>in the collection"] --> B["$match<br/>stock > 0"]
    B --> C["4 documents"]
    C --> D["$sort<br/>price desc"]
    D --> E["$limit 3"]
    E --> F["3 documents"]

Each stage receives the previous stage's output; the document count can only shrink here.

JS Watch the stream shrink and order change results

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the last case the sort does nothing, because after $project no document has a price field. The title order you see is just the incoming order. Real MongoDB behaves the same way: a stage only sees what the previous stage produced.

Using the output

A few habits make pipelines easy to read and debug:

  • Build it one stage at a time. Run the first stage alone, look at the output, then add the next. Every stage's input is the previous stage's output, so you can always check it.
  • Think in shapes. After each stage ask: what does one document look like now?
  • Do not expect a stable order without $sort. If you need an order, say so.
  • Use find when it is enough. A simple filter with a projection is shorter and just as fast; use a pipeline when you need to compute, group or join.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does each stage in an aggregation pipeline receive as its input?

    Stages run in order; the output of one stage is the input of the next. Only the first stage reads the collection.
  2. 2. You run [{ $project: { title: 1 } }, { $sort: { price: -1 } }]. What happens?

    After $project only title (and _id) remain, so there is nothing to sort by. Sort first or include price in the projection.
  3. 3. Why is it usually best to place $match early in a pipeline?

    Fewer documents flow downstream, and the first stage can be answered with an index like a find filter. The pipeline does not fail if $match comes later, though.
  4. 4. What does { $count: "n" } output when it receives no documents?

    $count emits no document at all for an empty stream, so check for an empty result instead of expecting a zero.

Summary

  • A pipeline is an ordered list of stages; each stage's output feeds the next one.
  • $match filters with the normal query language, $project shapes fields, $sort, $skip and $limit order and trim, and $count counts.
  • A stage only sees the fields the previous stage produced, so projecting too early can break later stages.
  • Put $match early to shrink the stream and to allow index use; $sort followed by $limit is cheap.
  • Build and test pipelines one stage at a time.

Next, summarize many documents into groups with Grouping with $group.