Skip to content
Tutorials Center
My learning

Topics

Lesson 105 of 149 ยท Data and the Web

Document Databases with MongoDB

Use MongoDB from Python with PyMongo. Learn documents and collections, CRUD, query and update operators, indexes and the aggregation pipeline.

Relational databases like SQLite, PostgreSQL and MySQL store data in tables with fixed columns. Document databases take a different approach: they store each record as a self-contained document, a nested structure that looks just like a Python dictionary or a JSON object. A customer document can hold the customer's addresses as a list and their preferences as a nested object, with no extra tables and no joins.

MongoDB is the most popular document database, and PyMongo is its official Python driver. In this lesson you'll learn how documents and collections work, run the basic operations (create, read, update, delete) with PyMongo, write queries with MongoDB's operators, speed them up with indexes, summarize data with the aggregation pipeline and decide how to shape your documents.

What runs here and what doesn't. A MongoDB server can't run in your browser, so all PyMongo code on this page is read-only, for you to try against a local server or a free cloud cluster. To let you practice the query model right here, the live examples use small helper functions written for this lesson that imitate how MongoDB matches, updates and groups documents. They are simplified teaching tools, not MongoDB itself.

Documents and collections

MongoDB organizes data in three levels. A server holds databases, a database holds collections, and a collection holds documents. If you know SQL, this table maps the words:

SQL MongoDB
database database
table collection
row document
column field
primary key the _id field
join embedded documents, or $lookup

Every document has an _id field that's unique within its collection. If you don't supply one, MongoDB generates an ObjectId, a 12-byte value that includes a timestamp. Documents are stored in BSON (binary JSON), which supports more types than JSON, such as real dates, 64-bit integers, decimals and binary data. In Python, documents are simply dictionaries.

The big difference from SQL is that documents in one collection don't have to share the same fields. That flexibility is useful, but it means your code must expect missing fields:

Python Documents are nested dictionaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

PyMongo converts Python types to BSON and back for you: dict, list, str, int, float, bool, None, datetime.datetime and bytes all work. Plain JSON can't store a datetime (default=str only turns it into text), but BSON can, so in MongoDB dates stay real dates you can sort and compare. One catch: BSON has no date-only type, so store a datetime.date as a datetime (or as an ISO string).

Connecting with PyMongo

Install the driver with pip install pymongo. For a local server, the quickest start is Docker: docker run -d -p 27017:27017 mongo. Then connect with a connection string:

Python
from pymongo import MongoClient

client = MongoClient("mongodb://localhost:27017/")   # one client per program
client.admin.command("ping")                         # raises if the server is unreachable

db = client["shop"]                  # databases and collections are created lazily,
products = db["products"]            # on the first insert

A MongoClient keeps its own connection pool and is safe to share between threads, so create it once and reuse it. Cloud clusters such as MongoDB Atlas use the mongodb+srv://user:[email protected]/ form of the URL; as with any database, read it from an environment variable instead of hard-coding the password.

CRUD with PyMongo

The four basic operations each come in a "one" and a "many" version, and each returns a small result object that tells you what happened:

Python
# Create
result = products.insert_one({"name": "Ceramic Mug", "price": 12.5, "tags": ["kitchen"]})
print(result.inserted_id)                         # an ObjectId such as 6650c2...
products.insert_many([{"name": "Desk Lamp", "price": 34.0},
                      {"name": "Notebook", "price": 6.9}])

# Read
mug = products.find_one({"name": "Ceramic Mug"})  # a dict, or None
for product in products.find({"price": {"$lt": 20}}, {"_id": 0, "name": 1, "price": 1})\
                       .sort("price", -1).limit(10):
    print(product)
print(products.count_documents({"tags": "kitchen"}))

# Update
result = products.update_one({"name": "Desk Lamp"}, {"$set": {"price": 29.0}})
print(result.matched_count, result.modified_count)
products.update_many({}, {"$inc": {"views": 1}})
products.update_one({"name": "Desk Fan"}, {"$set": {"price": 25.0}}, upsert=True)

# Delete
result = products.delete_many({"price": {"$gt": 100}})
print(result.deleted_count)

A few points worth knowing from day one:

  • find() returns a cursor that fetches results lazily in batches as you loop; .sort(), .skip() and .limit() are applied on the server.
  • The second argument to find() is a projection: 1 includes a field, 0 excludes it. _id is included unless you exclude it.
  • Updates need an update operator like $set. Passing a plain dictionary to update_one() raises an error, so you can't wipe a document by accident; replace_one() is the method that replaces a whole document.
  • upsert=True updates the matching document, or inserts a new one if nothing matches.
  • An _id generated by MongoDB is an ObjectId, not a string. To look up an id that arrived as text (say, from a URL), convert it: from bson import ObjectId and then find_one({"_id": ObjectId(text_id)}).

The query language

A MongoDB filter is itself a document. {"name": "Notebook"} means "name equals Notebook". For other comparisons, you use operators, keys that start with $:

Operator Meaning Example filter
$eq, $ne equal, not equal {"status": {"$ne": "archived"}}
$gt, $gte, $lt, $lte greater / less than {"price": {"$gte": 10, "$lt": 50}}
$in, $nin in / not in a list {"category": {"$in": ["office", "garden"]}}
$exists the field is present (or not) {"discount": {"$exists": True}}
$and, $or combine filters {"$or": [{"price": {"$lt": 5}}, {"tags": "sale"}]}
dotted path a field inside a nested document {"stock.store": {"$gt": 0}}

Several fields in one filter are combined with AND. Two rules surprise newcomers: a dotted path like "stock.store" reaches inside embedded documents, and a filter on an array field matches when any element matches, so {"tags": "gift"} finds every product whose tags list contains "gift".

The live example below implements a small matches() function with these rules, so you can experiment with real filter documents:

Python MongoDB-style filters on Python dicts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try your own filters at the bottom. For example, {"tags": {"$nin": ["office"]}} finds products that have no office tag, and {"stock.store": {"$exists": False}} finds products with no store stock recorded at all.

Update operators

Updates describe changes, not whole new documents. The most used operators are:

Operator Effect Example
$set set fields (creating them if needed) {"$set": {"price": 9.5, "stock.store": 4}}
$unset remove fields {"$unset": {"discount": ""}}
$inc add to a number (negative to subtract) {"$inc": {"stock.store": -1}}
$push append to an array {"$push": {"tags": "sale"}}
$addToSet append only if not already present {"$addToSet": {"tags": "gift"}}
$pull remove matching array elements {"$pull": {"tags": "sale"}}

Because the server applies each update to one document atomically, $inc is safe even when many users buy the last few items at once. Reading the value in Python, adding one and writing it back would not be. Here's a small imitation of update_one() so you can see what each operator does to a document:

Python Applying update operators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that $set on stock.warehouse created a new field inside the nested document, and $inc on sold created the field from zero. The real server behaves the same way.

Indexes

Without an index, MongoDB answers a query with a collection scan: it examines every document. That's fine for a few thousand documents and painful for millions. An index keeps the values of one or more fields in sorted order, pointing back to the documents, so the server can jump straight to the matches:

Python
from pymongo import ASCENDING, DESCENDING

products.create_index("sku", unique=True)                  # also rejects duplicate SKUs
products.create_index([("category", ASCENDING), ("price", DESCENDING)])  # compound
orders.create_index("created", expireAfterSeconds=60 * 60 * 24 * 30)    # TTL: auto-delete

plan = products.find({"category": "office"}).sort("price", -1).explain()
print(plan["queryPlanner"]["winningPlan"])                 # IXSCAN = index used, COLLSCAN = full scan
print(products.index_information())

The live example shows the difference in work, counting how many documents each approach has to look at. A Python dictionary plays the role of the index:

Python Collection scan versus index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Indexes aren't free: every insert and update must also update each index, and they use memory. Index the fields you filter and sort by often, and check with explain(). A field like email where almost every value is different benefits hugely; a field with only five values, like country here, helps far less on its own.

The aggregation pipeline

For reports (totals per category, averages per month, top customers), MongoDB has the aggregation pipeline: a list of stages, each transforming the stream of documents and passing the result to the next stage, much like a chain of generators:

Python
pipeline = [
    {"$match": {"status": "paid"}},                          # like WHERE
    {"$unwind": "$items"},                                   # one document per array item
    {"$group": {"_id": "$items.category",                    # like GROUP BY
                "revenue": {"$sum": {"$multiply": ["$items.qty", "$items.price"]}},
                "orders": {"$sum": 1}}},
    {"$sort": {"revenue": -1}},                              # like ORDER BY
    {"$limit": 3},
]
for row in db.orders.aggregate(pipeline):
    print(row)

Other common stages are $project (reshape documents), $lookup (join documents from another collection), $count and $facet. Inside expressions, "$field" with a dollar sign means "the value of this field". This live version runs the core stages on Python lists so you can follow the data from stage to stage:

Python A mini aggregation pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cancelled order disappeared at $match, $unwind turned 3 orders into 5 single-item documents, and $group collapsed them to one result per category. On a real server, put $match first so that an index can be used and later stages handle fewer documents.

Bulk writes and atomic find-and-modify

Calling insert_one() or update_one() in a loop sends one network round trip per document. For hundreds or thousands of changes, bulk_write() sends a whole batch of mixed operations at once. You describe each operation with a small request object and hand over the list:

Python
from pymongo import InsertOne, UpdateOne, DeleteOne, ReplaceOne
from pymongo.errors import BulkWriteError

operations = [
    InsertOne({"sku": "A-100", "price": 9.5, "stock": 20}),
    UpdateOne({"sku": "B-200"}, {"$inc": {"stock": -3}}),
    UpdateOne({"sku": "C-300"}, {"$set": {"price": 4.0}}, upsert=True),
    ReplaceOne({"sku": "D-400"}, {"sku": "D-400", "price": 15.0, "stock": 0}),
    DeleteOne({"sku": "OLD-1"}),
]

try:
    result = products.bulk_write(operations, ordered=False)
    print(result.inserted_count, result.matched_count, result.modified_count,
          result.upserted_count, result.deleted_count)
except BulkWriteError as error:
    print(error.details["writeErrors"])            # one entry per failed operation

The ordered flag decides what happens after a failure:

  • ordered=True (the default) runs the operations in order and stops at the first error. Later operations are never tried. Use it when each step depends on the one before.
  • ordered=False lets the server run the operations in any order and keeps going after errors, then reports all failures together in a BulkWriteError. It is usually faster and is the right choice for independent operations, such as importing a big file where a few rows may be duplicates.

A bulk write is not a transaction: operations that succeeded before an error stay applied. For simple imports, insert_many(docs, ordered=False) is the shorthand. The live example below imitates the two modes on a plain dictionary, so you can see exactly which operations survive a failure:

Python Ordered versus unordered bulk writes (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ordered mode stored only ids 1 and 2 and then stopped. Unordered mode skipped the single bad document and stored 1, 2, 3 and 4. A real server reports the same thing in BulkWriteError.details.

Reading and changing in one step

Sometimes you need to change a document and also get it back, with nothing able to slip in between. Reading it with find_one() and then writing with update_one() leaves a gap in which another process can grab the same document. MongoDB closes the gap with three atomic methods:

Method What it does
find_one_and_update(filter, update) changes the first matching document and returns it
find_one_and_replace(filter, replacement) replaces the first matching document and returns it
find_one_and_delete(filter) deletes the first matching document and returns it

By default they return the document as it was before the change. Pass return_document=ReturnDocument.AFTER to get the new version, and sort= to choose which match is taken when several qualify:

Python
from pymongo import ReturnDocument

# A job queue: claim the oldest waiting job, so two workers never get the same one.
job = db.jobs.find_one_and_update(
    {"status": "waiting"},
    {"$set": {"status": "running", "worker": "worker-7"}},
    sort=[("created", 1)],
    return_document=ReturnDocument.AFTER,
)
if job is None:
    print("queue is empty")

# An auto-increment counter that is safe under concurrency.
counter = db.counters.find_one_and_update(
    {"_id": "invoice"}, {"$inc": {"value": 1}},
    upsert=True, return_document=ReturnDocument.AFTER)
print("next invoice number:", counter["value"])

Because the server does the match and the change as one operation on one document, two workers calling this at the same moment each receive a different job. The live version below shows why the two-step approach fails. It simulates two workers whose "read" and "write" steps interleave:

Python Why claiming a job in two steps goes wrong (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the two-step claim, both workers read job 1 as "waiting" before either wrote, so they both ran it and job 2 was never touched. The atomic version hands out jobs 1 and 2 separately. The same idea explains why $inc beats "read, add, write back" for counters.

Compound indexes and the order of fields

Earlier you created a compound index on category and price. The order of the fields in a compound index matters. An index on (category, price) is sorted by category first, and by price only within each category. That makes it useful for:

  • queries on category alone (the leftmost field, called the index prefix),
  • queries on category and price together,
  • queries that filter on category and sort by price.

It does not help a query on price alone, because prices are scattered through every category. A good rule of thumb for ordering the fields is equality first, then sort, then range: put the fields you compare with equality first, then the field you sort by, then fields you compare with $gt or $lt.

Python
products.create_index([("category", 1), ("price", 1)])

products.find({"category": "office"})                       # uses the index (prefix)
products.find({"category": "office", "price": {"$lt": 20}})  # uses the index fully
products.find({"price": {"$lt": 20}})                       # cannot use it efficiently

# Index only the documents you query: smaller and cheaper to maintain.
products.create_index("discount", partialFilterExpression={"discount": {"$exists": True}})

A partial index (with partialFilterExpression) contains only documents that match a filter, so it stays small when most documents don't have the field. Use explain() to check: a stage named IXSCAN with your index's name means it was used, while COLLSCAN means a full scan.

More pipeline stages: $lookup, $project and $addFields

The pipeline you met earlier grouped orders. These stages complete a typical reporting query. $lookup joins another collection (a left outer join, like SQL's LEFT JOIN), $addFields adds computed fields, and $project chooses the fields in the result:

Python
pipeline = [
    {"$match": {"status": "paid"}},
    {"$lookup": {"from": "customers",             # the other collection
                 "localField": "customer_id",     # field in orders
                 "foreignField": "_id",           # field in customers
                 "as": "customer"}},              # the matches arrive as an array
    {"$unwind": "$customer"},                     # one customer per order
    {"$addFields": {"total": {"$sum": "$items.price"}}},
    {"$project": {"_id": 0, "order": "$_id", "name": "$customer.name", "total": 1}},
    {"$sort": {"total": -1}},
]
for row in db.orders.aggregate(pipeline):
    print(row)

Use $lookup sparingly. If you need the same join on every read, it's often a sign that the data should be embedded instead. This simulation joins two lists the way $lookup does, so you can see the array that arrives in the as field:

Python What $lookup produces (simulation)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Order 3 has no matching customer, so its customer array is empty and $unwind removes it. If you want to keep such orders, set "preserveNullAndEmptyArrays": True on the $unwind stage.

Modeling data: embed or reference?

The most important design question in MongoDB is whether related data goes inside a document or in a separate collection that you link by _id:

Embed whenโ€ฆ Reference whenโ€ฆ
the data is read together (an order and its items) the data is shared by many documents (a product used in many orders)
the child has no life of its own (a post's tags) the list can grow without limit (all comments ever on a busy page)
the embedded list stays small the related data changes often and independently

A rule of thumb: design documents around how your app reads data. An order page needs the order, its items and the shipping address together, so embed them. One read then gets everything. Keep a copy of the product name and price inside each order item, too, because the order must show what the customer actually paid, even after the product's price changes. Also remember that a single document is limited to 16 MB, so unbounded arrays belong in their own collection.

Security: operator injection

Parameters protect SQL from injection, but MongoDB has its own version of the problem. Filters are dictionaries, so if your code puts user-supplied JSON straight into a filter, an attacker can send an operator instead of a value:

Python Operator injection, and the fix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The crafted body turned "the password equals X" into "the password is not equal to X", which is true, so the attacker logged in as root without knowing the password. (Real apps compare password hashes in Python, but the same trick works on any filter.) The fix is to check types before building a filter: make sure values are strings, numbers or whatever you expect, never dictionaries. Libraries like Pydantic, or your web framework's validation, do this for you.

MongoDB or SQL?

MongoDB fits well when records are naturally nested and read as a whole, when different records have different fields (product catalogs, event logs, content management), or when you need to spread huge amounts of data across many servers. A relational database fits better when data is highly connected, when you need many different joins and reports, or when strict constraints across tables matter. Both can do more than their reputation suggests: MongoDB supports multi-document transactions (on replica sets), and PostgreSQL stores and indexes JSON documents in jsonb columns. Choose based on how your data is shaped and queried, not on fashion.

Check your understanding

Quick check ยท 8 questions

of 8 answered You scored / 8 ยท perfect!

  1. 1. Which filter finds products whose tags array contains "gift"?

    Matching a single value against an array field succeeds when any element matches. The first option matches only arrays that are exactly ["gift"].
  2. 2. What does products.update_one({"sku": "A1"}, {"$inc": {"stock": -1}}) do?

    $inc adds the given amount, and a negative amount subtracts. The server applies the change atomically, so concurrent updates don't lose counts.
  3. 3. Your route receives an id as the text "6650c2f1a3b4c5d6e7f80912" and find_one({"_id": text_id}) returns None, although the document exists. Why?

    A string never equals an ObjectId. Convert it with bson.ObjectId before querying.
  4. 4. In an aggregation pipeline, why should $match usually come first?

    Stages run in order. Filtering first lets MongoDB use indexes and reduces the work for every stage after it.
  5. 5. A login handler builds {"username": body["username"], "password": body["password"]} from request JSON. What is the risk?

    This is operator injection. Validate that each value has the expected type (for example, a string) before using it in a filter.
  6. 6. What does bulk_write(operations, ordered=False) do when one operation fails?

    Unordered mode continues after errors and reports them together. Ordered mode stops at the first error. Neither mode is a transaction, so earlier successes stay applied.
  7. 7. Two workers must never claim the same waiting job. Which call is the right tool?

    find_one_and_update matches and changes one document atomically, so a second worker can't match a job that the first one has already changed.
  8. 8. You have a compound index on (category, price). Which query can NOT use it efficiently?

    A compound index is sorted by its first field. Queries must use a prefix of the index fields, and price alone is not a prefix.

Summary

  • MongoDB stores documents (nested, JSON-like, BSON on disk) in collections; each document has a unique _id, often an ObjectId.

  • Documents in a collection can have different fields, so code should handle missing ones.

  • PyMongo: one MongoClient per program; insert_one/many, find_one, find (with projection, sort, limit), update_one/many, delete_one/many, count_documents.

  • Filters are documents with operators like $gt, $in, $exists and $or; dotted paths reach nested fields, and array fields match any element.

  • Updates use operators ($set, $inc, $push, $addToSet, $pull, $unset), applied atomically per document; upsert=True inserts when nothing matches.

  • Indexes avoid collection scans; check queries with explain().

  • The aggregation pipeline ($match, $unwind, $group, $sort, $lookup, โ€ฆ) builds reports on the server.

  • bulk_write() sends many mixed operations at once (InsertOne, UpdateOne, DeleteOne, ReplaceOne); ordered=False keeps going after errors. It is not a transaction.

  • find_one_and_update/replace/delete change and return a document atomically (return_document=ReturnDocument.AFTER, sort=), which makes job queues and counters safe.

  • Compound index field order matters: queries must use a prefix; put equality fields first, then sort, then range. Partial indexes stay small.

  • $lookup joins collections, $addFields and $project shape the output; use $lookup sparingly.

  • Embed data that's read together; reference data that's shared or grows without limit. Validate input types to prevent operator injection.

Next: Web Development with Flask and Django, where you'll see how Python serves websites and APIs built on top of databases like these.