Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Streams, TTL and Integrations

Integrations With Lambda, Kinesis and S3

Connect DynamoDB to AWS Lambda, EventBridge Pipes, Kinesis Data Streams, OpenSearch and S3 exports to build event-driven apps, search indexes and analytics.

DynamoDB is rarely the whole system. Orders must trigger emails and payments, products must appear in a search box, and analysts want to run SQL over years of history. DynamoDB itself answers only key-based questions, so the surrounding AWS services do the rest. This lesson maps the common integrations and shows how to wire them without losing events.

All service calls on this page are read-only examples. DynamoDB, Lambda and S3 are not available inside your browser, so runnable parts are stand-ins for the data flow.

The integration map

Goal Typical service How it connects
React to each change AWS Lambda event source mapping on the table's stream
Route changes with filters and transforms Amazon EventBridge Pipes stream as the source, any target
Long retention, many consumers Kinesis Data Streams the table streams to Kinesis
Full-text and fuzzy search Amazon OpenSearch Service zero-ETL integration, or a Lambda that indexes
Analytics with SQL S3 + Athena ExportTableToPointInTime or incremental export
Queue work SQS / SNS Lambda or Pipes target
Workflows Step Functions Pipes target
Cache DAX in-memory write-through cache in front of reads

From table to the world

Lambda Run code per change batch
EventBridge Pipes Filter, enrich, route
Kinesis Long retention, many readers
S3 export Analytics with Athena

Streams carry change; exports carry history.

1 / 2

Lambda event source mapping

An event source mapping polls the stream for you. Important settings:

  • BatchSize (up to 10,000) and MaximumBatchingWindowInSeconds to group records.
  • StartingPosition: LATEST or TRIM_HORIZON (oldest available).
  • ParallelizationFactor (up to 10): concurrent batches per shard, still in order per item.
  • FilterCriteria: only invoke for matching records (for example eventName = INSERT), so you pay for fewer invocations.
  • BisectBatchOnFunctionError, MaximumRetryAttempts, and an OnFailure destination to isolate poison records.
  • FunctionResponseTypes = ["ReportBatchItemFailures"] lets the function report which record failed so only that record and the ones after it are retried.
Python
# handler with partial batch response (read-only here)
def handler(event, context):
    failures = []
    for rec in event["Records"]:
        try:
            process(rec)
        except Exception:
            failures.append({"itemIdentifier": rec["dynamodb"]["SequenceNumber"]})
    return {"batchItemFailures": failures}

Because a failing record blocks later records of the same shard until retries are exhausted, always configure a failure destination, otherwise one bad item can stall the pipeline for up to 24 hours.

EventBridge Pipes

A pipe has a source (the stream), optional filtering, optional enrichment (Lambda, API call) and a target (SQS, SNS, EventBridge bus, Step Functions, Kinesis, API destination). It replaces glue Lambdas that only forward data. A filter pattern looks like:

JSON
{
  "eventName": ["INSERT"],
  "dynamodb": { "NewImage": { "Type": { "S": ["Order"] } } }
}

Kinesis Data Streams for DynamoDB

You can also stream table changes to a Kinesis data stream of your own. It gives retention up to 365 days, several independent consumers (Firehose to S3, Managed Service for Apache Flink, Lambda), and a copy of every change. DynamoDB Streams and Kinesis streaming can both be enabled, but each record is billed by its own service.

Search with OpenSearch

DynamoDB cannot search free text. Two ways:

  1. DynamoDB zero-ETL integration with OpenSearch Service, managed by an OpenSearch Ingestion pipeline. It does an initial load and then follows changes.
  2. A Lambda on the stream that indexes NEW_IMAGE and deletes on REMOVE.

The application queries OpenSearch for matching IDs and loads items from DynamoDB by key. The search index is eventually consistent with the table.

Analytics: export to S3

Querying a table with a Scan for reports is slow and expensive. Instead, export to S3 (full export from a point-in-time recovery snapshot, or an incremental export of a time window). Export uses no read capacity and writes DynamoDB JSON or Amazon Ion files. Query them with Athena or load them into a warehouse. An import from S3 can create a new table from files.

Terminal
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:eu-west-1:123456789012:table/Shop \
  --s3-bucket my-export-bucket --export-format DYNAMODB_JSON

Point-in-time recovery must be on to export. Backups are described in backups and global tables.

Try it: filter and route stream records

The stand-in below reproduces what a pipe does: apply a filter pattern, transform each record and send it to different targets.

Python A pipe: filter, transform, route

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Building a derived search view

Here a stand-in shows the pattern behind OpenSearch indexing: a stream handler maintains an inverted index of words to item IDs.

Python A stream handler maintains a search index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The handler is idempotent: processing the same record twice leaves the same index.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why configure an OnFailure destination on a stream event source mapping?

    A failing record blocks later records in the same shard. A failure destination plus bisecting and retry limits keep the pipeline moving.
  2. 2. What is the best way to run analytics SQL over a large table?

    Exports to S3 use no read capacity and produce files that Athena or a warehouse can query.
  3. 3. What do EventBridge Pipes add to a stream source?

    A pipe connects a source to a target with optional filter and enrichment steps.
  4. 4. How should an application use OpenSearch next to DynamoDB?

    The table stays the source of truth. The search index is a derived, eventually consistent view.

Summary

  • Lambda event source mappings, EventBridge Pipes and Kinesis Data Streams all consume table changes.
  • Use filters, partial-batch responses and failure destinations to keep stream processing healthy.
  • Search and analytics belong in OpenSearch and S3/Athena; DynamoDB stays the source of truth.
  • Exports to S3 need point-in-time recovery and use no read capacity.
  • Derived views are eventually consistent and handlers must be idempotent.

Next: Backups, PITR and Global Tables