Message Deduplication Pattern

Overview

Message brokers guarantee at-least-once delivery, which means consumers may receive the same message more than once. Network retries, consumer crashes during processing, and broker redelivery all cause duplicates. Without deduplication, a payment might be processed twice or a notification sent multiple times.

The Message Deduplication pattern tracks message IDs in a store. Before processing, the consumer checks if the message ID has already been handled. If yes, it skips processing. If no, it processes the message and records the ID.

When to Use

• Your message broker provides at-least-once delivery (most do: SQS, RabbitMQ, Kafka)
• Processing a message twice causes side effects (payments, emails, inventory changes)
• You need exactly-once processing semantics without a broker that supports it natively
• Consumer crashes happen and messages get redelivered
• You process webhooks from third-party services that may retry on network failures
• You consume from multiple queues and need cross-queue deduplication

When to Avoid

• Your broker provides exactly-once delivery natively. Kafka transactions and SQS FIFO with content-based dedup already handle this. Adding application-level dedup is redundant.
• Messages are idempotent by nature. If processing twice has no side effects (e.g., updating a last-seen timestamp), dedup adds overhead without value.
• Throughput is critical and every millisecond counts. Redis dedup adds ~0.5ms per message. For ultra-low-latency systems, rely on broker guarantees instead.
• You cannot afford a dedup store dependency. If Redis downtime is unacceptable and you cannot fall back to idempotent processing, reconsider the architecture.
• Messages have no natural unique ID. Generating content hashes for every message adds CPU overhead and may dedup incorrectly for same-payload-different-intent cases.

Solution

Python (Redis + SQS)

import redis
import json
import hashlib

r = redis.Redis(host="localhost", port=6379, db=0)
DEDUP_TTL = 86400  # 24 hours

def process_message(message_id, payload):
    # Check if already processed
    dedup_key = f"dedup:{message_id}"
    if r.exists(dedup_key):
        print(f"Skipping duplicate message {message_id}")
        return

    # Process the message
    result = handle_order(payload)

    # Mark as processed with TTL
    r.setex(dedup_key, DEDUP_TTL, "1")
    print(f"Processed message {message_id}")

def handle_order(payload):
    order = json.loads(payload)
    print(f"Charging payment for order {order['order_id']}")
    return {"status": "charged"}

# Simulate duplicate delivery
process_message("msg-001", '{"order_id": 42}')
process_message("msg-001", '{"order_id": 42}')  # Skipped

JavaScript (Redis + BullMQ)

import Redis from "ioredis";
import { Worker } from "bullmq";

const redis = new Redis({ host: "localhost", port: 6379 });
const DEDUP_TTL = 86400; // 24 hours

async function isDuplicate(messageId) {
  const key = `dedup:${messageId}`;
  const result = await redis.set(key, "1", "EX", DEDUP_TTL, "NX");
  // result is "OK" if the key was set (first time), null if it already existed
  return result === null;
}

const worker = new Worker(
  "orders",
  async (job) => {
    const messageId = job.data.messageId;
    const payload = job.data.payload;

    if (await isDuplicate(messageId)) {
      console.log(`Skipping duplicate message ${messageId}`);
      return { status: "skipped" };
    }

    // Process the message
    console.log(`Charging payment for order ${payload.orderId}`);
    return { status: "processed" };
  },
  { connection: { host: "localhost", port: 6379 } }
);

Java (Redis + Spring)

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.util.concurrent.TimeUnit;

@Component
public class DeduplicatingConsumer {

    private final StringRedisTemplate redis;
    private static final int DEDUP_TTL = 86400; // 24 hours

    public DeduplicatingConsumer(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public void processMessage(String messageId, String payload) {
        String dedupKey = "dedup:" + messageId;

        // Atomic check-and-set: returns true if key was set (first time)
        Boolean wasSet = redis.opsForValue()
            .setIfAbsent(dedupKey, "1", DEDUP_TTL, TimeUnit.SECONDS);

        if (Boolean.FALSE.equals(wasSet)) {
            System.out.println("Skipping duplicate message " + messageId);
            return;
        }

        // Process the message
        System.out.println("Processing message " + messageId + ": " + payload);
        handleOrder(payload);
    }

    private void handleOrder(String payload) {
        // Business logic here
    }
}

Explanation

The pattern uses an atomic check-and-set operation (Redis SET NX EX) to determine if a message has been processed. The operation is atomic: only one consumer can set the key, so concurrent consumers processing the same message will not both proceed.

The TTL on the deduplication key prevents the store from growing indefinitely. After the TTL expires, the same message ID could be processed again. Choose a TTL longer than your maximum redelivery window (typically 24 hours for SQS, or the queue’s retention period).

For the deduplication to work, each message must carry a unique identifier. This can be a content hash (for deduplication based on payload) or a producer-assigned ID (for deduplication based on identity).

Variants

Variant	Store	Use Case	Tradeoff
Redis SET NX	Redis	Fast, shared across consumers	External dependency, TTL-based expiry
Database Unique Constraint	SQL DB	Durable, transactional	Slower, adds DB load
Content Hashing	Any store	Dedup by payload content	Same payload always deduped, even if different intent
Broker Native	SQS FIFO	Built-in dedup	Only with FIFO queues, limited throughput
In-Memory Set	Process memory	Single consumer, fast	Lost on restart, not shared across instances

What Works

• Use atomic check-and-set (Redis SET NX EX) to avoid race conditions between concurrent consumers
• Set TTL longer than your broker’s redelivery window
• Use content hashes when messages lack explicit IDs
• Make consumers idempotent as a second layer of defense: even if dedup fails, processing twice should not cause harm
• Log skipped duplicates for debugging and monitoring
• Use FIFO queues with content-based deduplication when the broker supports it (SQS FIFO)

Common Mistakes

• Check-then-set without atomicity: Two consumers check simultaneously, both see no key, both process. Always use atomic SET NX.
• TTL too short: If the TTL expires before the broker stops redelivering, duplicates slip through. Set TTL to at least 2x the redelivery window.
• Using process memory for dedup: Lost on restart, not shared across consumer instances.
• Not handling dedup store failures: If Redis is down, dedup fails. Decide whether to process (risk duplicates) or reject (lose messages).
• Dedup key based on mutable fields: If the key includes fields that change, the same logical message gets different keys and is processed twice.

How It Works

1. Message arrives with unique ID: Each message carries a deduplication key — either a producer-assigned UUID or a content hash. The consumer extracts this key before processing.
2. Atomic check-and-set: The consumer attempts SET NX EX on the dedup key in a shared store (Redis). If the key does not exist, the consumer sets it and proceeds. If it exists, the message was already processed — skip it.
3. Process the message: Only the consumer that successfully set the key processes the message. Concurrent consumers with the same message ID fail the SET NX and skip.
4. TTL expiry: After the configured TTL, the key expires. This prevents the store from growing indefinitely. The TTL must exceed the broker’s maximum redelivery window.

The atomicity of SET NX is critical: without it, two consumers could both check, both see no key, and both process the same message.

Best Practices

• For a deeper guide, see Idempotent Consumer Pattern.

• Use producer-assigned IDs over content hashes. Content hashes dedup identical payloads, which may be wrong if the same payload represents different logical operations. Producer-assigned IDs dedup by identity, not content.

• Log skipped duplicates. When a consumer skips a duplicate, log the message ID and timestamp. This helps debug redelivery issues and measure duplicate rates.

• Monitor dedup hit rate. If 50% of messages are duplicates, something is wrong upstream — the producer is retrying too aggressively or the consumer is too slow to acknowledge.

• Use FIFO queues when available. SQS FIFO and Kafka partition keys provide ordering and dedup at the broker level, reducing the need for application-level dedup.

• Graceful degradation on dedup store failure. If Redis is down, decide between processing with idempotency or rejecting. Document the choice and alert on it.

Real-World Examples

Stripe Payment Webhooks

Stripe sends webhooks for payment events. Network failures cause Stripe to retry, delivering the same webhook multiple times. Stripe recommends using event IDs for deduplication: check the event ID in a store before processing. Without dedup, a single payment could trigger multiple order fulfillments.

SQS + Lambda Order Processing

An e-commerce platform uses SQS to trigger Lambda for order processing. SQS may redeliver messages if Lambda fails to acknowledge in time. The platform uses Redis SET NX with the order ID as the dedup key. Duplicate deliveries are skipped, preventing double-charging or double-shipping.

Kafka Consumer with Redis Dedup

A streaming pipeline consumes events from Kafka and writes to a database. Kafka’s at-least-once delivery means the same event may be consumed twice. The consumer checks Redis before writing to the database. This provides exactly-once semantics without Kafka transactions.

Troubleshooting

• Messages are lost on restart: persist messages before acknowledging.
• Consumer lags behind producer: scale consumers, increase prefetch, and partition the topic.
• Duplicate messages: design consumers to be idempotent.
• Ordering is wrong after scaling: preserve partition keys and avoid rebalancing during bursts. Consider a single partition when order is mandatory.
• Queue depth grows but consumers are idle: check network partitions, consumer health, and permission issues. Restart gracefully.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the deduplication and pattern guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply message deduplication pattern when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Common Production Pitfalls

• Applying the pattern where no abstraction is needed, adding accidental complexity.
• Letting the pattern leak into unrelated modules and blur ownership boundaries.
• Over-engineering the first implementation instead of starting simple and measuring pain.
• Skipping contract tests, so refactors silently break consumers.
• Ignoring failure modes that the pattern does not cover.
• Using the pattern as a default instead of choosing the right tool for the current scale.
• Forgetting to document when to stop using the pattern and what replaces it.
• Missing observability around the pattern’s performance and error propagation.