Message Queues and Asynchronous Processing
Queues decouple producers from consumers so work can be buffered, retried and scaled independently.
Why use a queue?
- Absorb traffic spikes (load levelling).
- Move slow work out of the request path (emails, video processing).
- Let services fail and recover independently.
- Fan out one event to many consumers.
Models
| Model | Behaviour | Examples |
|---|---|---|
| Point-to-point queue | Each message handled by one consumer | SQS, RabbitMQ queues |
| Publish-subscribe | Each subscriber gets a copy | SNS, Redis Pub/Sub |
| Log / stream | Ordered, durable, replayable partitions | Kafka, Kinesis |
Kafka concepts
Topic "orders"
Partition 0: [m0][m1][m2]... <- ordered within a partition
Partition 1: [m0][m1]...
Consumer group A: each partition read by one consumer in the group
Messages with the same key go to the same partition, which preserves per-key order.
Delivery guarantees
| Guarantee | Meaning |
|---|---|
| At most once | May lose messages, never duplicates |
| At least once | No loss, duplicates possible (most common) |
| Exactly once | Hard; usually "at least once + idempotent consumers" |
Idempotency
Make handlers safe to run twice, for example by storing processed message IDs.
idempotent.py
def handle(message, processed_ids, apply):
if message["id"] in processed_ids:
return # duplicate delivery: ignore
apply(message)
processed_ids.add(message["id"])
Failure handling
- Retries with exponential backoff and jitter.
- Dead-letter queue (DLQ) for messages that keep failing.
- Back-pressure: slow producers or shed load when consumers lag.
- Outbox pattern: write the event in the same DB transaction as the data, then publish it, to avoid losing events.
Loading Comments...