Kinesis Cheat Sheet

Overview

• Serverless (almost) stream processing
  • “Almost” because you still have to plan, manage and pay for shards
• Can stream data, video, store stream to file and do stream analytics

Data Stream

• Real-time streaming
• 1MB payload
• Stream partitioned by shards
• Exactly-once is not achieved by Kinesis itself because of: producer and consumer retries
  • Use a unique ID in message to handle duplicate messages in your application

Firehose

• Save stream data to S3
• Near-real-time in AWS definition (~60 seconds latency for buffering)
• Can do transformation of data before writing to target

Analytics

• Use SQL to analyze incoming data with a sliding window

Kinesis Client Library

• Written in Java and other languages
• Used by producer to send data to stream
• Data is buffered before sending to stream, thus making it non-real-time
  • Use Kinesis API (PutRecords) directly if real-time streaming is needed