Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Reading and Writing Items

BatchGetItem and BatchWriteItem

Read and write many items in fewer round trips with BatchGetItem and BatchWriteItem: limits, partial failures, unprocessed items, backoff and when to use transactions.

Every DynamoDB request is an HTTPS round trip. Fetching 100 items one at a time means 100 round trips, each with network latency and overhead. The batch operations bundle many keys or writes into one request, which cuts latency and request overhead. They do not make anything cheaper in capacity terms (each item still costs what it costs), and they come with a rule that surprises people: a batch can partly fail. This lesson shows how to use them correctly.

BatchGetItem

BatchGetItem reads up to 100 items or 16 MB per call, from one or more tables, by full primary key.

Terminal
aws dynamodb batch-get-item --request-items '{
  "Books": {
    "Keys": [
      {"Isbn": {"S": "978-1"}},
      {"Isbn": {"S": "978-2"}},
      {"Isbn": {"S": "978-3"}}
    ],
    "ProjectionExpression": "Isbn, Title",
    "ConsistentRead": false
  }
}'

Things to know:

  • Keys are full primary keys, like GetItem. It is not a query.
  • The results are returned in no particular order. Match them to your keys by key value.
  • Missing items are simply absent; there is no error.
  • The response may contain UnprocessedKeys: keys DynamoDB did not get to (response size limit reached, throttling). You must resend those.
  • Each table in the request can have its own projection and consistency setting.
  • Reads are executed in parallel under the hood, so it is usually faster than a loop of GetItem calls.

BatchWriteItem

BatchWriteItem sends up to 25 put or delete requests (16 MB total, each item up to 400 KB) across one or more tables.

Terminal
aws dynamodb batch-write-item --request-items '{
  "Books": [
    {"PutRequest":    {"Item": {"Isbn": {"S": "978-4"}, "Title": {"S": "Fourth"}}}},
    {"PutRequest":    {"Item": {"Isbn": {"S": "978-5"}, "Title": {"S": "Fifth"}}}},
    {"DeleteRequest": {"Key":  {"Isbn": {"S": "978-1"}}}}
  ]
}'

What it cannot do:

  • It cannot UpdateItem. Only whole-item puts and deletes.
  • It cannot carry condition expressions.
  • It cannot put and delete the same item in one request.
  • It is not atomic. Some requests can succeed and others fail. For all-or-nothing behavior use TransactWriteItems (100 items, higher cost).
  • It does not return old values.

Like reads, it can return UnprocessedItems, which you must resend.

Partial failure and backoff

When DynamoDB is throttling or a partition is busy it may process only part of a batch. The standard approach:

  1. Send the batch.
  2. If the response has unprocessed items or keys, wait, then resend only those.
  3. Increase the wait each time (exponential backoff), add random jitter so clients do not retry in lockstep, and stop after a sensible number of attempts.

The AWS SDKs retry whole failed requests with backoff, but unprocessed items are a successful response, so the SDK does not retry them for you. Several SDK helpers (for example batch_writer in Python's boto3 resource interface) do the chunking and re-sending for you.

Python
with table.batch_writer() as batch:
    for book in books:
        batch.put_item(Item=book)

Chunking: more than 25 or 100 items

Real workloads exceed the limits, so split your input.

Python Stand-in: chunk a list into batches of 25

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A batch writer with unprocessed items

This stand-in processes only some requests per call, the way a throttled partition can. The client keeps resending what comes back, with exponential backoff (the waits are computed and printed, not slept).

Python Stand-in: resend UnprocessedItems with backoff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Twenty-five requests with a capacity of ten per call need three attempts. A client that ignored UnprocessedItems would silently lose fifteen writes. That is the most common bug with batch operations.

Reading in batches

Here a batch read of keys that includes missing ones. Results come back unordered, so the code builds a lookup by key.

Python Stand-in: BatchGetItem returns unordered, skips missing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Batch versus transaction versus individual calls

Need Use
Many reads by key, no atomicity BatchGetItem
Many puts/deletes, speed matters, partial failure acceptable BatchWriteItem
Several writes that must all succeed or all fail TransactWriteItems
Updates or conditions on each item Individual calls in parallel, or a transaction
Loading millions of items Import from S3, or parallel batch writers

If you just need to run many independent calls fast, running individual UpdateItem or GetItem calls in parallel threads is often as fast as batching and gives you per-item errors and conditions.

Common mistakes

  • Ignoring UnprocessedItems / UnprocessedKeys. Data is lost without any exception.
  • Retrying immediately in a tight loop. It worsens throttling; use backoff with jitter.
  • Assuming order. Batch reads return results unordered.
  • Expecting atomicity. A batch write can leave you with a half-applied state.
  • Sending duplicate keys in one batch write, which is rejected.
  • Trying to update with BatchWriteItem. It only puts whole items or deletes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the maximum number of put or delete requests in one BatchWriteItem call?

    BatchWriteItem accepts at most 25 requests (BatchGetItem allows up to 100 keys).
  2. 2. What must a client do when the response contains UnprocessedItems?

    Unprocessed items are a successful response, so the client is responsible for resending them, ideally with exponential backoff and jitter.
  3. 3. Which is true of BatchWriteItem?

    Batch writes support only PutRequest and DeleteRequest, with no conditions and no atomicity.
  4. 4. In what order does BatchGetItem return items?

    Results are unordered, so match items to your keys by their key attributes.

Summary

  • BatchGetItem reads up to 100 items; BatchWriteItem writes or deletes up to 25.
  • Batches reduce round trips but not capacity cost, and they can partly fail.
  • Always resend UnprocessedKeys and UnprocessedItems, using exponential backoff with jitter.
  • Batch writes cannot update, cannot be conditional and are not atomic; use transactions for all-or-nothing.
  • Chunk your data, and match batch read results to keys yourself.

Next: The Query Operation