Task queues, pub/sub fan-out, event streaming, and request-reply are different problems. Picking the tool by brand name before naming the problem costs real money in migrations and monthly bills. The comparison below walks through Kafka, RabbitMQ, and SQS on the criteria that decide production outcomes: throughput, ordering, operational cost, and the use case your event-driven architecture serves.

Kafka: a distributed log with replay

Kafka stores events in a distributed log that consumers can replay from the beginning, and consumer groups scale by adding members, and partition counts are fixed at topic creation, so sizing decisions deserve thought before launch. Throughput reaches hundreds of thousands of messages per second on modest clusters. The price shows up in operations: brokers, partitions, rebalances, and retention demand attention even on MSK or Confluent. Event sourcing, analytics pipelines, and broadcast to several independent consumers are where it earns its keep.

RabbitMQ: a smart broker for work queues

RabbitMQ bets on rich routing inside the broker: topic exchanges filter by key patterns, dead-letter exchanges capture rejected messages, priorities reorder deliveries, quorum queues replicate state across nodes. It handles tens of thousands of messages per second with headroom and covers task queues with complex topology using fewer moving parts than a Kafka cluster.

SQS: serverless queues inside AWS

SQS charges per message, scales on its own, and demands zero ops hours. Known limits: 256KB per message, a 15-minute maximum delay, pricing in cents per million requests. FIFO queues add strict ordering with throughput ceilings (300 req/s default, up to 3,000 with batching). AWS-native teams get queue infrastructure without managing anything.

Ordering: the detail that sinks designs

Kafka guarantees order within a partition, so the partition key decides everything: orders partitioned by customer_id preserve sequence per customer. RabbitMQ also drops global ordering under competing consumers. Design idempotent consumers regardless of the tool; infrastructure ordering optimizes, idempotency corrects.

Operational cost side by side

  • Managed Kafka (MSK, Confluent): starts near $200+/month for the smallest cluster
  • Self-hosted RabbitMQ: cheap instances, but HA, failover, and upgrades land on your backlog
  • SQS: pay-per-message pricing, scales to zero when traffic disappears, zero servers to babysit

Where to start and when to migrate

Start with SQS when the team lives in AWS, or RabbitMQ when rich routing matters outside it. Graduate to Kafka when history replay, multiple independent consumers, or 100k+ msg/s become real requirements. A small team running a job queue almost never needs its own cluster; it needs a reliable queue with decent observability.