Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Queries, Scans and Indexes

The Scan Operation and Parallel Scan

Learn what Scan does, why it is expensive, when it is acceptable, how parallel scan with segments works, and the better alternatives: indexes, exports and design changes.

Scan reads every item in a table (or index), in no meaningful order, and optionally filters the result. Where Query jumps to one partition, Scan walks them all. It is the operation that makes DynamoDB look like a slow, costly database when people use it as a general SELECT * WHERE .... Used for the right jobs it is fine, and this lesson separates those from the traps.

How Scan works

Terminal
aws dynamodb scan \
  --table-name Books \
  --filter-expression "Pages > :p" \
  --expression-attribute-values '{":p": {"N": "300"}}'
  • It reads items in 1 MB pages and returns LastEvaluatedKey for continuation, just like Query. You loop until it is absent.
  • The FilterExpression is applied after reading, so you are charged for every item read, however few are returned. See filters and pagination.
  • Projections reduce the response size, not the cost.
  • Eventually consistent by default, with half the read cost. ConsistentRead=true is available and doubles it.
  • The order of results follows internal partition order, not the order you would expect. Never rely on it.

A scan that reads a 50 GB table costs the capacity of reading 50 GB, every time you run it, and competes with your normal traffic for throughput on provisioned tables.

Why Scan is a trap

The cost grows linearly with table size, while latency and spend for an application request should not. A feature that scans looks great with 1,000 rows in development and breaks at 100 million. Typical misuses:

  • searching for an item by a non-key attribute ("find the user with this email"),
  • listing "all active orders",
  • counting items,
  • sorting or top-N by an attribute.

All of those should be answered by an index, a key design, a counter or another service.

When Scan is fine

  • Small tables (settings, feature flags, lookup tables of a few hundred items).
  • One-off jobs: migrations, backfills, data repair, exports, audits.
  • Batch analytics run off-peak, often better done against an export.
  • Local development and tests with tiny data.

Even then, limit the load: use a smaller page size, pause between pages, or run it at night.

Parallel scan

A single scan is sequential. To read a large table faster you can split it into segments. You choose TotalSegments (say 8) and run 8 workers, worker n passing Segment = n. DynamoDB divides the key space so the segments cover the table without overlap. All workers run at once, each with its own pagination state.

Terminal
aws dynamodb scan --table-name Books --total-segments 8 --segment 0
aws dynamodb scan --table-name Books --total-segments 8 --segment 1
# ... up to segment 7

More segments finish sooner but consume capacity faster: 8 workers read about 8 times the units per second, so on a provisioned table a parallel scan can throttle your production traffic. Throttle the scan, or run it against a restored copy or an export.

flowchart TB
    T["Table"] --> S0["Segment 0<br/>worker 0"]
    T --> S1["Segment 1<br/>worker 1"]
    T --> S2["Segment 2<br/>worker 2"]
    T --> S3["Segment 3<br/>worker 3"]
    S0 --> M["Merge results"]
    S1 --> M
    S2 --> M
    S3 --> M

A model of scan and parallel scan

The stand-in assigns items to segments by hash and runs a scan per segment. It shows that segments are disjoint, that together they cover everything, and what a filter does to the read count.

Python Stand-in: segmented scan covers the table once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The filter returned few items, but every item was read. Splitting into segments made the work parallel, not smaller.

Scan versus Query in cost

Python Stand-in: reads needed to find one customer's orders

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both return the same five items, but one costs over a thousand times more. The gap widens as the table grows.

Alternatives to scanning

Need Better approach
Find an item by another attribute A global secondary index keyed on that attribute
List items of a type or status A sparse index, or a key design that groups them
Count items A counter item, or Streams plus an aggregate
Analytics over the whole table Export to S3 (point-in-time, no capacity used), then Athena or Spark
Full-text search OpenSearch, fed from a stream
Copy or migrate a table Export/import, or a throttled parallel scan

Being a good citizen when you must scan

  • Use a modest Limit (page size) and add a short pause between pages.
  • Use eventually consistent reads.
  • Project only the attributes you need.
  • Run during quiet hours, or with a rate limiter tied to your provisioned capacity.
  • Watch ConsumedCapacity (request ReturnConsumedCapacity=TOTAL) and back off on throttling.
Python Stand-in: a throttled full scan

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Using Scan on the request path. It works in testing and fails at scale.
  • Believing a filter makes it cheap.
  • Forgetting to paginate, so only the first megabyte is processed.
  • Parallel scans on a busy provisioned table causing throttling of real users.
  • Counting by scanning. Keep a counter.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does Scan read?

    Scan reads the whole table in 1 MB pages. The filter only trims what is returned.
  2. 2. What does parallel scan with TotalSegments=8 do?

    Each worker passes its own Segment number. The slices cover the table once, but total capacity consumed does not shrink.
  3. 3. Which is an acceptable use of Scan?

    One-off jobs and small tables are fine. The others should use an index, a key design or a counter.
  4. 4. How can you analyze a whole large table without using its capacity?

    Point-in-time export to S3 reads from backups, so it uses no table read capacity.

Summary

  • Scan reads the whole table in 1 MB pages; filters do not reduce what you pay for.
  • Cost and latency grow with table size, so avoid it on application paths.
  • Fine uses: small tables, one-off jobs, throttled maintenance.
  • Parallel scan with Segment and TotalSegments speeds up large reads but consumes capacity faster.
  • Prefer indexes, key design, counters and S3 exports.

Next: Global Secondary Indexes