Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Queries, Scans and Indexes

Global Secondary Indexes (GSI)

Add alternate query paths with global secondary indexes: key schema, projection types, eventual consistency, separate capacity, sparse indexes, backfill and costs.

A table's primary key gives you one fast access path. Real applications need more. You look up a user by id but also by email. You list orders by customer but also by status. Without another route, each of these means a scan. A global secondary index (GSI) is that route: an automatically maintained copy of part of your table, organized under a different partition key and optional sort key, which you can Query like a small table of its own.

What a GSI is

  • A GSI has its own key schema: a partition key and, optionally, a sort key, on any attribute whose type is string, number or binary.
  • It stores a copy of items from the base table, with the attributes you choose to project.
  • DynamoDB keeps it in step with the table asynchronously. You never write to an index directly.
  • It is "global" because it spans all partitions of the base table (unlike a local secondary index, which shares the base partition key).
  • Its keys do not need to be unique. Many base items can have the same index key.
  • It has its own throughput (on provisioned tables) and its own storage cost.
flowchart LR
    W["Write to base table"] --> T["Base table<br/>PK: UserId"]
    T -. "async copy" .-> G["GSI: EmailIndex<br/>PK: Email"]
    Q1["GetItem / Query by UserId"] --> T
    Q2["Query by Email"] --> G

Creating an index

You can define GSIs when you create the table or add them later.

Terminal
aws dynamodb update-table \
  --table-name Users \
  --attribute-definitions \
      AttributeName=Email,AttributeType=S \
      AttributeName=CreatedAt,AttributeType=S \
  --global-secondary-index-updates '[{
    "Create": {
      "IndexName": "EmailIndex",
      "KeySchema": [
        {"AttributeName": "Email", "KeyType": "HASH"},
        {"AttributeName": "CreatedAt", "KeyType": "RANGE"}
      ],
      "Projection": {"ProjectionType": "INCLUDE", "NonKeyAttributes": ["Name", "Plan"]}
    }
  }]'

Adding an index to an existing table triggers a backfill: DynamoDB reads every item and builds the index, with the index in CREATING status until done. Large tables can take a long time, and the index cannot serve queries until it is ACTIVE.

Querying an index

You use the same Query operation with IndexName:

Terminal
aws dynamodb query --table-name Users --index-name EmailIndex \
  --key-condition-expression "Email = :e" \
  --expression-attribute-values '{":e": {"S": "[email protected]"}}'

The key condition rules are the same as for a table: equality on the index's partition key, optional condition on its sort key.

Projection types

Each index stores some attributes of the base item. The index always includes the base table's primary key and the index key attributes. You decide about the rest:

Projection Stores Use it when
KEYS_ONLY Only table and index keys You can fetch the full item afterwards with GetItem, or you only need keys
INCLUDE Keys plus a list you name You read a few specific attributes from the index
ALL Every attribute You always need the whole item and accept the extra storage and write cost

A query on an index can only return projected attributes, unless it fetches attributes from the base table, which a GSI does not do automatically (unlike a local secondary index). If you need more, either project more, or take the keys and call GetItem / BatchGetItem.

Projection is a cost decision: every projected attribute is stored twice and written twice. Project only what the index's queries need.

Consistency

GSI reads are eventually consistent only. ConsistentRead is not supported. A write to the table propagates to the index normally within a fraction of a second, but there is no guarantee. Design flows where "I just wrote this and need to find it through the index immediately" use the base table or tolerate a short delay.

Sparse indexes

If an item lacks the index's key attribute, it is not written to the index at all. That makes a "sparse" index: it contains only items that have the attribute. Use it to index a small subset of a large table, for example only orders that need attention.

Give the attribute only to items that should be indexed (NeedsReviewAt), and remove it when done. The index stays tiny and queries over "all items needing review" are cheap.

Write amplification and throttling

Every table write that changes an indexed or projected attribute also writes to the index, which consumes write capacity on the index. On provisioned tables, an under-provisioned GSI can throttle writes to the base table. With on-demand tables this is automatic, but you still pay for each index write. Rule of thumb: every GSI adds roughly one extra write per item write, so add them with care.

A GSI model with projection and sparsity

This stand-in keeps a base table and an index in sync, supports KEYS_ONLY, INCLUDE and ALL, and leaves out items without the key attribute.

Python Stand-in: a GSI with projection and sparse keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The email index returned Name but not Plan because Plan was not projected, and two items with the same email both appear (GSI keys are not unique). The review index holds just one of three items.

Using a GSI to enforce uniqueness is a mistake

Because GSI keys are not unique and updates are asynchronous, an index cannot enforce a unique constraint. Use a separate lookup item with a conditional write for uniqueness, as in choosing partition keys.

Reverse lookup and many-to-many

A very common GSI is the inverted index: swap the partition and sort key of the table. If the table has PK = USER#1, SK = GROUP#7 (user belongs to group), the index with PK = SK and SK = PK answers "which users are in group 7?" with a single query.

Python Stand-in: inverted index for many-to-many

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical guidance

  • Start from your access patterns; add an index only for one you cannot serve with the table keys.
  • Choose a high-cardinality index partition key. Index partitions get hot like table partitions do.
  • Project as little as the queries need.
  • Expect a delay between writing and seeing a change in the index.
  • Remember the limit of 20 GSIs per table by default, and plan names clearly.
  • To change an index key, create a new index, switch your code, then delete the old one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which reads does a GSI support?

    GSIs are updated asynchronously, so reads from them are only eventually consistent.
  2. 2. What happens to a base item that lacks the GSI's partition key attribute?

    Items without the index key attribute are left out, which is what makes sparse indexes possible.
  3. 3. Which projection stores only the table and index keys?

    KEYS_ONLY keeps the keys; INCLUDE adds chosen attributes and ALL copies everything.
  4. 4. Are GSI keys unique?

    An index key identifies a group of items, not a single item. Index entries are distinguished by the base table key.

Summary

  • A GSI is an automatically maintained copy of table data under a different key, queried with IndexName.
  • Index keys need not be unique; the base table key is always included.
  • Projection (KEYS_ONLY, INCLUDE, ALL) controls what the index stores, what you can read and what it costs.
  • GSIs are eventually consistent, have their own capacity and add write cost.
  • Items without the key attribute are left out, giving sparse indexes.

Next: Local Secondary Indexes (LSI)