Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 45 ยท Storage and Databases

Firestore and Bigtable

Understand Firestore documents, collections and indexes, and Bigtable wide-column tables and row-key design, with a document-store stand-in you can query.

Not every application fits neatly into rows and joins. Google Cloud has two popular NoSQL databases that solve different problems. Firestore stores flexible JSON-like documents and is built for apps that need quick lookups, live updates and automatic scaling. Bigtable is a wide-column database designed for enormous volumes of data, such as time series, with very low latency at huge throughput.

This lesson explains the data model of each, how indexes and consistency behave, and how to design a Bigtable row key. Limits, prices and features change, so check the current documentation before designing something real.

Firestore: documents and collections

The Firestore model

Document A set of fields, like a small JSON object.
Collection A named group of documents.
Subcollection A collection nested inside a document.

Paths alternate: collection, document, collection, document.

1 / 2

A document is identified by a path such as users/ada/orders/1001. It holds fields of types like string, number, boolean, timestamp, array, map (a nested object) and reference. A document has a size limit (around 1 MiB), so you do not store unbounded lists inside it; you use subcollections or separate documents instead.

Firestore has two modes. Native mode is the full-featured one with real-time listeners and mobile/web SDKs. Datastore mode keeps the older Datastore API for server-side apps. Choose when you create the database; the mode cannot be changed afterwards.

Python
# Firestore client library (read-only: needs a real project and credentials)
from google.cloud import firestore

db = firestore.Client()
db.collection("users").document("ada").set({"name": "Ada", "city": "London"})

doc = db.collection("users").document("ada").get()
print(doc.to_dict())

cheap = (
    db.collection("products")
    .where("price", "<", 20)
    .order_by("price")
    .limit(5)
    .stream()
)

Datastore mode in practice

Cloud Datastore was Google's original serverless NoSQL database. It now runs on the Firestore engine, and a Firestore database in Datastore mode keeps its API: data is organized as entities (documents) of a kind (collection), identified by a key, with properties of simple types, and optional ancestor paths that group related entities for strongly consistent queries. It suits server-side applications, especially older App Engine apps, that need automatic scaling with no servers to run. Good habits:

  • Avoid a very high rate of writes to one key range or to a single entity group, which creates a hotspot.
  • Do not use sequential numeric IDs for high write rates. Let Datastore allocate IDs, or use random ones.
  • Keep indexes to the queries you run: every indexed property costs write time and storage. Exclude large or rarely queried properties from indexing.
  • Use batch operations and transactions for related changes, and keep entities small.

Queries and indexes

Firestore only runs queries it can answer by reading an index, so reads stay fast no matter how large the collection is. Simple single-field indexes are created automatically. A query that filters or sorts on several fields needs a composite index, which you define yourself; if it is missing, the error message includes a link to create it. Because queries read only the matching index entries, cost and time depend on the number of results, not the collection size. The flip side: Firestore has no joins, so you design documents around the screens and questions your app needs, often duplicating a little data (denormalizing) to avoid extra reads.

Consistency and transactions

Reads of a document or query return the latest committed data (strong consistency). You can group writes in a batch (all or nothing) or a transaction that reads and writes atomically, which is how you safely increment a counter or move money between two documents. Very hot single documents (updated many times per second) hit write limits, so spread such counters across several documents.

Bigtable: wide-column at scale

Bigtable stores a giant sorted map. A table has rows, each identified by one row key. Columns are grouped into column families, and each cell can hold several timestamped versions. Rows are stored sorted by row key, and a table is split into contiguous chunks called tablets that are spread across nodes.

Key facts:

  • You can read or write a single row by key, or scan a range of rows by key prefix, very fast. There are no secondary indexes and no joins.
  • Writes to a single row are atomic. There are no multi-row transactions.
  • You add nodes to increase throughput; performance depends heavily on how evenly the load spreads across keys.
  • It suits time series, IoT readings, ad-tech events, financial ticks and other write-heavy data measured in terabytes and beyond. It is a poor fit for small datasets (the cluster has a baseline cost) or for flexible ad-hoc queries.
Terminal
# cbt, the Bigtable command-line tool (read-only example)
cbt createtable sensor_readings
cbt createfamily sensor_readings metrics
cbt set sensor_readings sensor42#2025-01-15T10:00 metrics:temp=21.5
cbt read sensor_readings prefix=sensor42#2025-01-15

Row-key design

The row key is the only way to find data, so it is the most important design decision.

  • Put what you query by first. Reading "all readings of sensor 42 on one day" is a cheap prefix scan if the key is sensor42#2025-01-15T10:00.
  • Avoid keys that start with a timestamp or an ever-increasing number. All new writes would land on the same tablet and one node would be overloaded (a hotspot), while the others idle.
  • Separate fields with a delimiter such as #. Use fixed-width or zero-padded numbers so sorting works as text.
  • For "newest first" scans, use a reversed timestamp (for example a big number minus the time).
  • Keep keys short; they are stored with every row.

Inside a Bigtable table

The "giant sorted map" has a few more details that explain its behavior.

  • Cells and timestamps. A cell is the value at one row, one column family and one column qualifier. Each cell can keep several versions, each with a timestamp, and a garbage-collection policy per family says how many versions or how much age to keep. A read normally returns the newest version.
  • Column families and qualifiers. Families are declared up front and should be few. Qualifiers (the column names inside a family) are not declared: you invent them when you write. A row can have thousands of qualifiers, and one row can have different qualifiers from the next.
  • Sparse storage. Because empty cells take no space, a table with a million possible columns where each row fills ten costs only the ten. This is why it suits wide, irregular data such as per-user event counters.
  • Compression. Data is compressed in storage, and repeating values in sorted neighbors compress well.
  • Nodes and tablets. A table is divided by key range into tablets. A node is compute that serves tablets, and it does not hold the data itself: the data lives in Google's distributed file system (Colossus, see Cloud Storage). Adding a node moves tablet responsibility, not data, so rebalancing is fast. Each node answers reads and writes for its tablets and splits or merges busy or small ones.
  • Mutations, updates and deletes. A write is a mutation applied atomically to one row. An update just writes a new cell version. A delete writes a tombstone marker for a cell, a family or a row, and the old data is physically removed later.
  • Compaction. New writes go to memory and a log first, then are flushed as sorted immutable files. In the background Bigtable compacts those files, merging them, dropping deleted and expired versions and reducing the number of files a read must check. This is why deleting data does not shrink storage immediately and why reads are fast after compaction.

Compared with self-managed HBase, whose data model it follows, Bigtable removes much of the operations work (no cluster to patch, no region servers to balance by hand), scales by changing the node count without a bottleneck master, and replicates across clusters with a setting. The price is a managed service bill and an API that is Google-specific, though HBase-compatible clients exist.

Python A tiny sparse, versioned table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing between them

Question Firestore Bigtable
Typical data size Megabytes to terabytes Terabytes to petabytes
Access By path, rich queries with indexes By row key or key range
Real-time client updates Yes No
Throughput focus Moderate, auto scaling Very high, you size the nodes
Good for Apps, user profiles, chat, catalogs Time series, telemetry, analytics feeds

For relational data with joins, see Cloud SQL; the full decision guide is in Spanner, AlloyDB and Choosing a Database.

Stand-ins you can run

The real databases cannot run here. The first stand-in is a tiny in-memory document store with collections, where filters, ordering and limits, to show how document queries feel. It has no indexes, so it scans everything; Firestore answers from indexes instead.

JS A tiny document store

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second stand-in explains row-key hotspots. It spreads 12 sensors writing at consecutive times across 4 imaginary tablets (key ranges). With a timestamp-first key, the newest writes pile onto the last tablet. With a sensor-first key, they spread out.

JS Row-key hotspots

Press Run (or Ctrl + Enter) to see the console output here.

No output.

What is real and what is the stand-in: the Firestore and Bigtable concepts, the Python client call and the cbt commands describe real products but are read-only here. The JavaScript above is a toy model. The tablet mapping in the second example is invented purely for illustration; real Bigtable splits tablets automatically by observed load.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a Firestore query that sorts and filters on several fields sometimes fail?

    Firestore answers queries from indexes. Multi-field queries need a composite index, and the error message links to creating it.
  2. 2. Which Bigtable row key most likely creates a hotspot?

    Increasing keys send every new write to the last tablet, overloading one node while the rest sit idle.
  3. 3. Which workload fits Bigtable best?

    Bigtable is built for huge, write-heavy datasets accessed by row key. The chat app fits Firestore; joins fit a relational database.
  4. 4. What is a document in Firestore?

    Documents are flexible, JSON-like records at paths such as users/ada. Collections group them.

Summary

  • Firestore stores documents in collections, answers queries from indexes, scales automatically and can push live updates to clients.
  • Design Firestore data around your queries; there are no joins, so some duplication is normal.
  • Bigtable is a sorted wide-column store for huge, high-throughput data, accessed by row key or key range.
  • Row-key design decides Bigtable performance: lead with what you query by and avoid monotonically increasing prefixes.
  • Pick Firestore for app data and live sync, Bigtable for telemetry-scale workloads.

Next: Spanner, AlloyDB and Choosing a Database