Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 70 ยท Serverless and messaging

Kinesis Data Streams, Firehose and Managed Flink

Learn Amazon Kinesis: shards, partition keys, producers and consumers, retention, Firehose delivery to S3, Managed Service for Apache Flink, and Kinesis vs SQS and SNS.

A queue such as SQS hands each message to one worker, then deletes it. That is perfect for jobs. But some data is a stream: click events from a website, readings from thousands of sensors, log lines from a fleet of servers. The events arrive constantly and in order, several different programs want to read the same events, and sometimes you want to replay what happened an hour ago.

Amazon Kinesis is the AWS family for streaming data. This lesson covers three members: Kinesis Data Streams (the durable, ordered log), Amazon Data Firehose (the easy way to deliver a stream into S3 and other stores), and Managed Service for Apache Flink (continuous analysis of the stream). Kinesis Video Streams, for camera and audio feeds, is a separate service that we leave out.

Kinesis cannot run on this site. The CLI and boto3 snippets are real, read-only examples. The Python programs are stand-ins that model shard routing and Firehose buffering with plain Python, so you can see the behavior without an AWS account.

Kinesis Data Streams: shards and partition keys

A data stream is an append-only log split into one or more shards. A record has a partition key (a string you choose), a data blob of up to 1 MiB, and a sequence number that Kinesis assigns. The partition key is hashed, and the hash decides which shard stores the record. Records with the same key always land on the same shard, and order is guaranteed within a shard, not across the whole stream.

A shard has fixed capacity: up to 1 MB per second or 1,000 records per second for writes. For reads, a shard supports 2 MB per second shared among classic consumers. If you need more, you add shards (resharding), so a stream's throughput is the number of shards times those limits.

flowchart LR
    P1["Producer: app servers"] --> S
    P2["Producer: IoT devices"] --> S
    subgraph S["Data stream"]
        SH1["Shard 1"]
        SH2["Shard 2"]
        SH3["Shard 3"]
    end
    S --> C1["Consumer: Lambda"]
    S --> C2["Consumer: Flink app"]
    S --> C3["Firehose to S3"]

Several producers write, several independent consumers read the same records.

You choose how capacity is managed:

  • On-demand mode: Kinesis scales the shards for you and bills for the data written and read. It is the easy default for unpredictable traffic.
  • Provisioned mode: you set the number of shards and pay per shard-hour. It is cheaper for a steady, well-understood load, and you must reshard when traffic grows.

Retention and replay

Records stay in the stream for 24 hours by default, and you can extend retention up to 365 days. Reading a record does not remove it. Each consumer tracks its own position, so a new consumer can start from the oldest record (TRIM_HORIZON) or from a time in the past, and an existing one can rewind after a bug. This replay ability is the biggest difference from a queue.

Producers and consumers

Producers can call the API directly, use the Kinesis Producer Library, or use an agent. A boto3 producer is only a few lines (read-only):

Python
import boto3, json

kinesis = boto3.client("kinesis", region_name="eu-west-1")
kinesis.put_record(
    StreamName="clickstream",
    Data=json.dumps({"user": "u-42", "page": "/pricing"}).encode(),
    PartitionKey="u-42",          # same user -> same shard -> ordered
)

Consumers can be:

  • Lambda, through an event source mapping. Lambda reads each shard in batches and processes the records in order. A failing batch blocks that shard until it succeeds or expires, so use retry limits and an on-failure destination.
  • The Kinesis Client Library (KCL), which coordinates several workers and keeps checkpoints.
  • Enhanced fan-out: each registered consumer gets its own dedicated 2 MB per second per shard pipe, pushed over HTTP/2, so consumers do not compete for the shared read limit.
  • Firehose and Managed Flink, described below.

A hot shard

Because the key decides the shard, a bad key choice sends most traffic to one shard. A key such as country with 90% of users in one country makes a hot shard: it throttles while others sit idle. Choose a key with many evenly used values, such as a user or device ID.

This stand-in hashes keys with MD5 (as Kinesis does) and maps them to shard ranges. The details of the real mapping differ, but the effect on balance is the same.

Python Partition keys and shard balance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Amazon Data Firehose

Often you do not want to write a consumer. You just want the stream delivered somewhere: an S3 bucket for a data lake, Redshift, OpenSearch, Splunk or an HTTP endpoint. Data Firehose (formerly Kinesis Data Firehose) does this as a fully managed service with no shards to manage.

Firehose buffers incoming records until a size (for example 5 MB) or a time (for example 60 seconds) threshold is reached, then writes one object. This is why delivery is "near real time", not instant. Along the way it can:

  • run a Lambda transformation to clean or enrich each record,
  • convert JSON to Parquet or ORC for cheaper, faster queries with Athena,
  • compress the data and add a time-based prefix such as year=2026/month=11/day=10/,
  • save failed records to a separate S3 prefix.

A source can be a Kinesis data stream, a direct PutRecord call, or other AWS services such as CloudWatch Logs.

Python Firehose-style buffering into S3 objects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each S3 object is written when either limit is reached first. Many tiny objects make later queries slow, so larger buffers are better for analytics, and smaller buffers are better for freshness.

Capacity modes and typical use cases

A data stream runs in one of two capacity modes. In provisioned mode you choose the number of shards and pay per shard-hour, which is cheaper for a steady, predictable load but needs you to reshard when traffic grows. In on-demand mode Kinesis scales the stream for you and you pay per data volume, which suits unpredictable or spiky traffic and removes shard planning. You can switch a stream between modes.

The same family also has Kinesis Video Streams, which ingests video and audio from cameras and devices for storage, playback and machine-learning analysis (for example with Rekognition), and supports WebRTC for live, low-latency two-way video between devices and apps.

What teams build on streams:

  • Fraud detection: score each payment as it arrives and block or flag suspicious ones before the money moves.
  • Real-time dashboards and metrics: clickstream, game telemetry and IoT readings aggregated within seconds.
  • Personalized recommendations and marketing: update a customer's profile as events arrive and react while they are still on the site.
  • Log and event pipelines: gather application logs from many sources, then deliver them to S3, OpenSearch or a warehouse with Firehose.
  • Replayable event history: because records are kept for 24 hours to a year, a new consumer can read old data from a chosen position, which a plain queue cannot do.

Counting events per minute, finding the top pages of the last five minutes, or joining a stream with reference data are continuous queries: they run forever and update their answer as records arrive. Managed Service for Apache Flink (the successor of Kinesis Data Analytics) runs Apache Flink applications for you. You write the application in Java, Python or SQL, point it at a Kinesis stream (or Kafka), and AWS handles servers, scaling and checkpoints. Use it for windowed aggregations, anomaly detection and real-time dashboards. If your analysis is simple, a Lambda consumer is cheaper and easier.

Kinesis, SQS and SNS compared

SQS SNS Kinesis Data Streams
Model Queue Publish/subscribe Ordered, replayable log
Readers One worker per message Every subscriber gets a copy Many consumers, each with its own position
After reading Message is deleted Delivered, not stored Stays until retention ends
Ordering FIFO queues only FIFO topics only Per shard, by partition key
Replay No No Yes
Best for Background jobs Fan-out notifications Event streams and analytics

Rules of thumb: choose SQS for work to be done once, SNS to notify many systems, and Kinesis when order, replay or several independent readers of the same data matter. Kinesis shines at high volumes of small events. See SQS and SNS for the other two.

Cost thinking

You pay for the stream capacity (on-demand per GB, or provisioned per shard-hour), for extended retention beyond 24 hours, and for enhanced fan-out. Firehose charges per GB ingested, with extra for format conversion. Flink charges per processing unit-hour. Provisioned shards that are mostly idle are the classic waste: check the stream metrics in CloudWatch, such as WriteProvisionedThroughputExceeded for too few shards and low IncomingBytes for too many.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What decides which shard a record is stored in?

    Kinesis hashes the partition key and uses the result to pick a shard. Records with the same key go to the same shard, which preserves their order.
  2. 2. A stream has a hot shard. What is the most likely cause?

    Uneven partition keys concentrate traffic on one shard. A key with many evenly used values, such as a user ID, spreads the load.
  3. 3. What makes Kinesis different from SQS in how reading works?

    A stream is a log. Reading does not remove records, so consumers have their own positions and can replay data until retention expires.
  4. 4. Which service delivers a stream to S3 with buffering, compression and optional Parquet conversion, without writing a consumer?

    Firehose is the managed delivery service. It buffers by size or time, can transform records with Lambda, and writes objects to S3.
  5. 5. What is the default retention of a Kinesis data stream?

    The default is 24 hours. You can extend it up to 365 days, at extra cost.

Summary

  • Kinesis Data Streams is an ordered, replayable log split into shards. The partition key decides the shard, and order is guaranteed per shard.
  • Each shard accepts about 1 MB per second of writes. On-demand mode scales for you, provisioned mode is cheaper for steady loads.
  • Retention is 24 hours by default and up to 365 days. Consumers keep their own positions, and enhanced fan-out gives each one dedicated read throughput.
  • Firehose delivers a stream to S3, Redshift, OpenSearch and more, buffering by size or time and optionally converting to Parquet.
  • Managed Flink runs continuous queries over streams.
  • Use SQS for jobs, SNS for fan-out notifications and Kinesis for ordered, replayable event streams.

Next: EventBridge and Step Functions