Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Streams, TTL and Integrations

DynamoDB Streams

Capture every insert, update and delete in a DynamoDB table with Streams: view types, shards, ordering guarantees, 24-hour retention and common consumer patterns.

Many features sit next to a database rather than inside it: send a welcome email when a user is created, update a search index when a product changes, keep a counter per category, write an audit trail. If your application writes to DynamoDB and then calls these systems itself, a crash between the two steps loses the second one. DynamoDB Streams moves that responsibility to the database. Every change to an item is recorded in an ordered log, and consumers read it at their own pace.

A stream is a change data capture (CDC) feed: a time-ordered sequence of item-level modifications that stays available for 24 hours. This lesson explains what is in a record, how ordering works and how to design consumers.

Turning a stream on

Streams are enabled per table, with a view type that decides what each record contains:

StreamViewType Record contains
KEYS_ONLY only the key attributes of the changed item
NEW_IMAGE the item as it looks after the change
OLD_IMAGE the item as it looked before the change
NEW_AND_OLD_IMAGES both, so you can compute what changed
Terminal
# read-only CLI example
aws dynamodb update-table --table-name Shop \
  --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES

A record has an eventName (INSERT, MODIFY or REMOVE), the keys, the images according to the view type, a sequence number and an approximate creation time. Reading from the stream is free of read capacity charges when you use the Lambda trigger; direct reads with the Streams API are billed as stream read request units.

What a stream gives you

Ordered log Per item, in write order
24 hours Records are kept one day
Images Old, new, both or keys only
At least once Consumers must be idempotent

Exactly once per record, with no duplicates, is not promised.

1 / 2

Ordering, shards and delivery

A stream is split into shards, each holding an ordered run of records for a set of partition keys. The guarantees are:

  • Changes to the same item appear in order and appear exactly once in the stream.
  • There is no order across different items in different shards.
  • Consumers deliver records at least once. The consumer may see a record again after a retry, so handlers must be idempotent (running twice has the same effect as once).
  • Records are available for 24 hours. A consumer that is down longer than that loses data.
  • The table's TTL deletes show up as REMOVE events with a userIdentity marking the DynamoDB service; see Time to Live.

Reading a stream

Two ways to consume:

  1. AWS Lambda trigger (event source mapping). The simplest. Lambda polls the shards, batches records and invokes your function. Configure batch size, batch window, retries, a bisect-on-error option and an on-failure destination (SQS or SNS) so a poison record does not block a shard forever. See stream integrations.
  2. Kinesis Data Streams for DynamoDB, an alternative that sends changes to a Kinesis stream (retention up to a year, many consumers, Firehose delivery).

The older DynamoDB Streams Kinesis Adapter and the low-level GetShardIterator/GetRecords API still exist, but they need you to track shards and checkpoints yourself.

A Lambda handler in Python (read-only here; Lambda and DynamoDB are not available in the browser):

Python
def handler(event, context):
    for record in event["Records"]:
        name = record["eventName"]                      # INSERT | MODIFY | REMOVE
        new = record["dynamodb"].get("NewImage")
        old = record["dynamodb"].get("OldImage")
        if name == "INSERT" and new["Type"]["S"] == "Order":
            send_confirmation(new["PK"]["S"])
    return {"batchItemFailures": []}

Note the stream image uses DynamoDB JSON: each value is wrapped in a type descriptor ({"S": "..."}, {"N": "10"}).

Try it: replay a change log

This stand-in table appends a record to a log whenever you write, shaped like a stream record with both images. It is not the real service.

Python Capture changes into a stream log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A MODIFY record with both images lets a consumer see that only plan changed, for example to trigger an upgrade email.

Consumer patterns

  • Derived data: copy items to OpenSearch for full-text search, to S3 for analytics, or to a cache.
  • Aggregation: keep a counter item per category. Be careful: a handler that does ADD count 1 is not idempotent. Store the sequence number processed or use a conditional update.
  • Audit and history: write each change to an append-only history table or to S3.
  • Fan-out and events: forward to EventBridge Pipes, SNS, SQS or Step Functions.
  • Cross-table consistency: maintain a denormalized copy (see adjacency lists).

Idempotent processing

Python At-least-once delivery needs idempotence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Per shard, a consumer can remember the last sequence number it applied. In a real handler you store that value together with the counter in one conditional update.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How long does DynamoDB Streams keep records?

    Stream records are available for 24 hours. For longer retention use Kinesis Data Streams for DynamoDB.
  2. 2. Which stream view type lets you compare an item before and after a change?

    NEW_AND_OLD_IMAGES includes both the pre-change and post-change item.
  3. 3. What ordering is guaranteed?

    Changes to one item are in order. Different items can be in different shards with no cross-item order.
  4. 4. Why must stream handlers be idempotent?

    Delivery is at least once. A repeat must not double a counter or send a second email.

Summary

  • A stream is an ordered, 24-hour log of item changes (INSERT, MODIFY, REMOVE).
  • The view type controls whether records hold keys only, the new image, the old image or both.
  • Order is guaranteed per item; delivery is at least once, so make consumers idempotent.
  • Lambda triggers are the easiest consumer; Kinesis Data Streams give longer retention.
  • Use streams for derived data, aggregation, audit and event fan-out.

Next: Time to Live (TTL)