Kinesis Data Streams and Firehose to S3 — Ingestion vs Stream Processing

Build an end-to-end streaming data pipeline on AWS using Kinesis Data Streams and Firehose. You will create a Kinesis Data Stream, write and read records with the AWS CLI, then set up a Firehose delivery stream that reads from the Data Stream, transforms records through a Lambda function, and delivers the results to S3.

You'll learn to:

• Create a Kinesis Data Stream and produce records with put-record
• Consume records using get-shard-iterator and get-records
• Create a Firehose delivery stream that reads from a Kinesis Data Stream
• Build a Lambda transformation that enriches and re-encodes records
• Verify delivered objects in S3 after Firehose buffer flush
• Understand when to use Data Streams (custom consumers, replay) vs Firehose (managed delivery)

Key Resources:

• Kinesis Data Streams Developer Guide
• Firehose Developer Guide
• Firehose Data Transformation