Message Queues

Why queues?

Temporal decoupling, retries, backpressure, ordering control.

Delivery semantics

At‑least‑once (default): requires idempotent consumers.
At‑most‑once: no retries → risk of loss.
Exactly‑once: expensive; emulate via idempotence + transactions.

Tools

Kafka (distributed log), RabbitMQ (AMQP), SQS/Pub/Sub (cloud).

Best practices

DLQ, metrics (lag, throughput), compaction (Kafka), balanced partitions.

Code: Producer/Consumer (Kafka, Node.js)

// language-javascript
import { Kafka } from 'kafkajs'
const kafka = new Kafka({ clientId: 'app', brokers: ['kafka:9092'] })

// Producer with idempotent key
const producer = kafka.producer()
await producer.connect()
await producer.send({
  topic: 'events',
  messages: [{ key: orderId, value: JSON.stringify(event) }],
})

// Consumer with at-least-once, idempotent handler
const consumer = kafka.consumer({ groupId: 'events-group' })
await consumer.connect()
await consumer.subscribe({ topic: 'events', fromBeginning: false })
await consumer.run({
  eachMessage: async ({ message }) => {
    const id = message.key.toString()
    if (!(await redis.setnx(`evt:${id}`, 1))) return // dedupe
    await handle(JSON.parse(message.value.toString()))
  },
})

Exactly-once discussion

True exactly-once is extremely hard in distributed systems. Aim for at-least-once with idempotent consumers, or transactional outbox + consumer offsets in the same DB transaction where possible.

Backpressure and redrive

# language-nginx
location /ingest {
  limit_req zone=ingest burst=100 nodelay;
  proxy_pass http://ingestor;
}

# language-bash
# Redrive failed messages from DLQ to main topic after fix
aws sqs receive-message --queue-url "$DLQ" | jq -r '.Messages[].ReceiptHandle' | while read h; do
  aws sqs send-message --queue-url "$MAIN" --message-body @payload.json
  aws sqs delete-message --queue-url "$DLQ" --receipt-handle "$h"
done

Ordering and partitioning

If ordering matters per key, choose partition key carefully (e.g., userId) and keep consumer group size aligned with partitions.

*Message Queues