Dead Letter Queues: Setup for SQS, RabbitMQ & Kafka

Overview

Dead letter queues (DLQs) capture messages that fail processing after repeated attempts in event-driven architectures. Without one, a malformed message either blocks a queue partition forever or disappears silently. A well-run DLQ setup separates poison messages (payloads that will never succeed) from transient failures, so operators can inspect, replay, or discard failed messages without touching the main flow.

The examples below configure a working DLQ on SQS, RabbitMQ, and Kafka, and the rest of the recipe covers the operational practices that keep a DLQ useful instead of a graveyard.

When to Use

Use this resource when:

• Consumers hit unrecoverable errors such as malformed payloads, schema mismatches, or missing references.
• One bad message could block a partition or every message queued behind it.
• Operations needs visibility into failed messages for manual review or compliance audit trails.
• You need a ceiling on retries so transient failures don’t loop forever.

A DLQ is overkill when message loss is acceptable and the broker already redelivers at low cost (presence pings, non-critical metrics). It also adds little when processing is idempotent and cheap enough that an unconstrained retry costs less than operating a second queue.

Solution

Amazon SQS — redrive policy (AWS CLI)

# Create the main queue and its DLQ
aws sqs create-queue --queue-name orders-queue
aws sqs create-queue --queue-name orders-dlq

# Resolve URLs and the DLQ ARN
QUEUE_URL=$(aws sqs get-queue-url --queue-name orders-queue --query 'QueueUrl' --output text)
DLQ_URL=$(aws sqs get-queue-url --queue-name orders-dlq --query 'QueueUrl' --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

# Send a message to the DLQ after 3 failed receives
aws sqs set-queue-attributes \
  --queue-url "$QUEUE_URL" \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"'$DLQ_ARN'\",\"maxReceiveCount\":3}"
  }'

RabbitMQ — dead-letter exchange (Python + pika)

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Dead-letter exchange (DLX) and its queue
channel.exchange_declare(exchange='orders.dlx', exchange_type='direct')
channel.queue_declare(queue='orders-dlq', durable=True)
channel.queue_bind(queue='orders-dlq', exchange='orders.dlx', routing_key='failed')

# Main queue routes rejected or expired messages to the DLX
args = {
    'x-dead-letter-exchange': 'orders.dlx',
    'x-dead-letter-routing-key': 'failed',
    'x-message-ttl': 300000  # 5 minutes
}
channel.queue_declare(queue='orders', durable=True, arguments=args)

def handle_message(ch, method, properties, body):
    try:
        process_order(body)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except UnrecoverableError:
        # requeue=False sends the message to the DLX
        ch.basic_reject(delivery_tag=method.delivery_tag, requeue=False)

channel.basic_consume(queue='orders', on_message_callback=handle_message)
channel.start_consuming()

Kafka — dead-letter topic (Node.js + KafkaJS)

Kafka has no built-in DLQ; the consumer publishes to a separate topic after retries are exhausted.

const { Kafka } = require('kafkajs');

async function main() {
  const kafka = new Kafka({ brokers: ['localhost:9092'] });
  const consumer = kafka.consumer({ groupId: 'order-processors' });
  const producer = kafka.producer();

  await consumer.connect();
  await producer.connect();
  await consumer.subscribe({ topic: 'orders', fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      try {
        await processOrder(JSON.parse(message.value.toString()));
      } catch (err) {
        // Publish to the DLQ with enough context to debug and replay
        await producer.send({
          topic: 'orders-dlq',
          messages: [{
            key: message.key,
            value: message.value,
            headers: {
              'error.type': err.name,
              'error.message': err.message,
              'original.topic': topic,
              'original.partition': String(partition),
              'original.offset': String(message.offset),
              'retry.count': '3'
            }
          }]
        });
      }
    }
  });
}

main().catch(console.error);

Companion code: the dead-letter-queue companion repo has these examples as runnable files.

Explanation

A message lands in the DLQ when the broker or the consumer decides it can no longer be delivered to the main flow. Each broker triggers that decision differently.

Condition	When to DLQ	Action
Max retries exceeded	After N failed attempts	Move to DLQ
Unparseable message	Invalid JSON, schema mismatch	Move immediately
Missing dependency	Referenced record doesn’t exist	Retry, then DLQ
Business rule violation	Order for non-existent product	Move immediately

Retrying a poison message is wasted work, so the unrecoverable cases should skip retries entirely and go straight to the DLQ. In RabbitMQ that’s basic_reject with requeue=False; in SQS it’s the maxReceiveCount on the redrive policy; in Kafka it’s your own consumer logic, because the broker knows nothing about processing failures.

Monitor the DLQ as a first-class metric. Alert when depth crosses a threshold (10 messages is a reasonable start) and when the oldest message ages past your incident SLA. A growing DLQ is a leading indicator of a systemic problem, not a queue-management detail. Attach the original trace ID to each DLQ message so you can follow it back through distributed tracing and correlate the failure with the deployment or dependency that caused it.

One subtlety worth knowing on Kafka: publishing to a DLQ topic commits the source offset, so later messages in the partition keep flowing. If you need per-key ordering, pause that key’s stream or restore order in the replay logic instead of assuming the DLQ preserves sequence.

Variants

Broker	DLQ mechanism	Configuration
AWS SQS	Redrive policy	maxReceiveCount + target ARN
RabbitMQ	Dead-letter exchange	x-dead-letter-exchange
Kafka	Consumer-managed	Separate topic + producer logic
Azure Service Bus	Built-in DLQ	MaxDeliveryCount exceeded, TTL expiry, or explicit dead-letter
Google Pub/Sub	Dead-letter topic	deadLetterPolicy.maxDeliveryAttempts

SQS and Azure Service Bus handle dead-lettering at the broker level; RabbitMQ routes through an exchange you declare; Kafka leaves everything to client code. For a RabbitMQ-only walkthrough with TTL and max-length triggers, see the RabbitMQ dead-letter queue recipe. For the design-pattern view of the same idea, see the Dead Letter Channel pattern.

What Works

• Set a real retry ceiling: 3–5 attempts balances recovery time against queue pressure; pair it with exponential backoff so transient failures drain before the DLQ sees them.
• Stamp DLQ messages with context: original queue or topic, error type, retry count, and trace ID. The KafkaJS example shows the header pattern; without it, operators can’t tell “bad JSON” from “database down”.
• Split DLQs by failure class: schema-invalid messages should never be replayed, while dependency failures should be replayed once the dependency recovers. Separate DLQs keep those replay semantics clean.
• Alert on depth and age, not existence: the first message in a DLQ is normal; a queue that keeps growing is the signal that matters. Show it in the same dashboards that carry your metrics collection so the signal doesn’t get buried.
• Replay in batches after the fix: republish a handful of messages, watch the error rate, then drain the rest. Blind bulk replay re-triggers the original failure at scale.

Common Mistakes

1. No DLQ at all: failed messages silently disappear or block consumers forever.
2. Infinite requeue: requeueing without a max count creates perpetual processing and can stall an entire partition.
3. An unmonitored DLQ: left alone it becomes a graveyard rather than a safety net; nobody looks at it until the audit asks.
4. Missing failure metadata: without error type and retry count, every DLQ triage starts from zero.
5. One shared DLQ for every topic: a poison message from a low-value topic mixes with critical failures and breaks per-topic replay.
6. Replaying without fixing the root cause: the same error comes back, now multiplied across every replayed message.
7. Treating the DLQ as a retry queue: a DLQ holds messages that already exhausted retries; if you want scheduled redelivery, that’s a retry queue with backoff, a different mechanism.

Operating the DLQ

• Inspection tooling: give ops a way to read DLQ messages without consuming them: body, headers, original queue, failure reason, and timestamp, filterable by error type or date. The SQS console and the RabbitMQ Management UI (port 15672) both support this; Kafka needs a UI tool or a small consumer that peeks without committing.
• Replay pipeline: build a script or admin action that republishes DLQ messages back to the source queue. Support selective replay by error type or date range, include a dry-run mode, and log every replay event for audit. Before requeueing, validate each message against the schema currently in production. Replaying a message that predates a schema change just re-poisons it.
• Retention: expire DLQ messages after a period longer than your incident SLA (7–14 days is typical). If compliance requires an audit trail, archive expired messages to cold storage before deletion.
• Multi-level DLQs: route transient failures (timeouts, connection drops) to a retryable DLQ and permanent failures (validation, schema) to a terminal one. Each gets different replay rules and alert thresholds.
• Sizing and circuit breaking: plan for roughly 1–5% of volume in the DLQ under normal conditions. A sustained spike beyond that is a signal to pause consumers or trip a circuit breaker, not to keep absorbing failures. The runbook for DLQ incidents covers the operational procedure.

Further Reading

• Amazon SQS dead-letter queues — official redrive policy reference.
• RabbitMQ dead-letter exchanges — official DLX documentation.
• Azure Service Bus dead-letter queues — built-in DLQ semantics.
• Google Pub/Sub dead-letter topics — deadLetterPolicy reference.
• Dead Letter Channel — Enterprise Integration Patterns — the original pattern description.
• Dead letter queue runbook — operational template for inspection, root-cause analysis, and replay.