SQS vs. Kinesis Data Stream Cheat Sheet

SQSKinesis Data Stream
Service TypeQueueStream
Service ManagementServerlessManaged, but user needs to manage shards
Number of consumersSingle; messages received by a consumer are made invisible for a duration to prevent from being processed by other consumersMultiple; for example one processor and one archiver
Message IngestionIngest any amount of messages; scales transparentlyEach shard provides 1MB data / 1000 messages ingestion
Message DeletionManual; if not delete the message will appear again after visibility timeoutNot possible; user can move
Message ReceptionAlways receive most recent messages in the whole queue; no limit to readsRead messages from a selected shard using a cursor; user has to maintain cursor manually; 5 reads per shard per second
Message OrderLoose order for standard queues; Strict order for FIFO queuesYes, strictly ordered by sequence number within a shard
Message Size<= 256KB<= 1MB
Individual Message DelayYes; up to 15 minutesNo
ReplayNo, messages are invisible or deleted from queue after receivedYes, messages remains in the stream, you just need to set the cursor to early position and process from there; could be used for off-hour analysis and auditing
FilteringNo, only a single consumer to receive all messages; create two queues for different processing needsYes, user can deliver message to different shards using partition keys
Ease of useEasy API, automatic scalingComplicated API, or use KCL for some automation
BillingBy number of requestsBy Shard-hour + data retention time + in-stream data size

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s