Claim Check Pattern

Overview

The Claim Check Pattern stores a large payload in external storage and sends only a lightweight reference token through the messaging infrastructure. The consumer uses that token to fetch the payload on its own side, so the broker never carries the heavy data.

Brokers enforce size limits for a reason: SQS caps messages at 256KB and Kafka defaults to about 1MB. A generated report, a video file or a bulk export can blow past those limits, and raising the limit usually trades one problem for a worse one.

When to Use

Use the Claim Check Pattern when:

• Payloads exceed the broker’s size limit and raising it isn’t an option — managed services like SQS don’t let you.
• Large messages slow the broker down or eat too much of its memory and disk.
• Several consumers need to read the same large payload; each can fetch it from storage instead of receiving a copy through the bus.
• Producers generate data faster than the pipeline drains it and you don’t want a huge payload blocking every message behind it.

When to Avoid

Skip it when:

• Payloads fit comfortably inside the broker’s limits — the extra hop to storage only adds latency and a failure point.
• The cost of external storage (requests, egress, retention cleanup) outweighs what you save on the broker.
• The consumer needs the payload fast; claim check adds a second network round-trip per message.
• There’s no reliable object storage (S3, Azure Blob, GCS) or the payload and the message must commit atomically in one transaction.

Why Brokers Struggle with Large Messages

Message brokers are built to route small envelopes fast, not to move files. A 20MB payload hurts in several ways at once: it sits in broker memory or on its log while every consumer catches up, it delays everything queued behind it (head-of-line blocking), and on replicated brokers like Kafka every byte is copied to followers, so a big message multiplies storage and network cost across the cluster. Producers pay for it too — a publish that takes seconds is a natural source of timeouts, retries and downstream duplicates.

Size limits exist to protect the broker from exactly this. SQS rejects anything over 256KB, Kafka refuses messages above message.max.bytes (1MB by default), and even brokers that accept larger payloads, like RabbitMQ at 16MB, degrade well before the limit. When the payload is bigger than the envelope, the envelope should carry a pointer instead. The message queues guide compares how RabbitMQ, Kafka and SQS each degrade as payloads grow.

Solution

Python

import uuid
import json
from typing import Optional
from datetime import datetime

# ============================================================================
# EXTERNAL STORAGE (simulating S3 / Azure Blob / GCS)
# ============================================================================

class BlobStorage:
    """In-memory blob store for demonstration"""
    def __init__(self):
        self._store = {}

    def upload(self, data: bytes, content_type: str = "application/json") -> str:
        token = str(uuid.uuid4())
        self._store[token] = {
            "data": data,
            "content_type": content_type,
            "created_at": datetime.now()
        }
        return token

    def download(self, token: str) -> Optional[bytes]:
        entry = self._store.get(token)
        return entry["data"] if entry else None

    def delete(self, token: str):
        self._store.pop(token, None)

# ============================================================================
# MESSAGE BROKER (simulating RabbitMQ / Kafka / SQS)
# ============================================================================

class MessageBroker:
    def __init__(self):
        self._queues = {}

    def publish(self, queue: str, message: dict):
        self._queues.setdefault(queue, []).append(message)

    def consume(self, queue: str) -> Optional[dict]:
        messages = self._queues.get(queue, [])
        return messages.pop(0) if messages else None

# ============================================================================
# CLAIM CHECK IMPLEMENTATION
# ============================================================================

class ClaimCheckProducer:
    """Publishes messages with claim check tokens instead of large payloads"""
    def __init__(self, broker: MessageBroker, storage: BlobStorage):
        self.broker = broker
        self.storage = storage

    def publish_large_message(self, queue: str, payload: dict, metadata: Optional[dict] = None):
        # Store payload in external storage
        payload_bytes = json.dumps(payload).encode("utf-8")
        token = self.storage.upload(payload_bytes, "application/json")

        # Publish lightweight message with claim check reference
        message = {
            "claim_check": token,
            "metadata": metadata or {},
            "payload_size": len(payload_bytes),
            "timestamp": datetime.now().isoformat()
        }
        self.broker.publish(queue, message)
        print(f"Published claim check: {token} ({len(payload_bytes)} bytes)")
        return token

class ClaimCheckConsumer:
    """Consumes messages and retrieves full payloads via claim check"""
    def __init__(self, broker: MessageBroker, storage: BlobStorage):
        self.broker = broker
        self.storage = storage

    def process_next(self, queue: str):
        message = self.broker.consume(queue)
        if not message:
            return None

        token = message["claim_check"]
        payload_bytes = self.storage.download(token)

        if payload_bytes is None:
            print(f"ERROR: Claim check {token} not found in storage")
            return None

        payload = json.loads(payload_bytes.decode("utf-8"))

        # Process the full payload
        print(f"Retrieved payload ({message['payload_size']} bytes): {payload['report_id']}")

        # Clean up stored payload after processing
        self.storage.delete(token)
        print(f"Deleted claim check: {token}")

        return payload


# ============================================================================
# USAGE
# ============================================================================

storage = BlobStorage()
broker = MessageBroker()

producer = ClaimCheckProducer(broker, storage)
consumer = ClaimCheckConsumer(broker, storage)

# Large payload: a detailed report that would exceed typical broker limits
large_report = {
    "report_id": "RPT-2024-Q1",
    "generated_at": datetime.now().isoformat(),
    "records": [
        {"id": i, "data": "x" * 1000} for i in range(1000)  # 1MB+ payload
    ],
    "summary": {"total": 1000, "revenue": 500000.00}
}

producer.publish_large_message("reports.queue", large_report, {"priority": "high"})
consumer.process_next("reports.queue")

Java

import java.util.*;
import java.util.concurrent.*;

// Blob storage abstraction
interface BlobStorage {
    String upload(byte[] data, String contentType);
    byte[] download(String token);
    void delete(String token);
}

class InMemoryBlobStorage implements BlobStorage {
    private final Map<String, byte[]> store = new ConcurrentHashMap<>();

    public String upload(byte[] data, String contentType) {
        String token = UUID.randomUUID().toString();
        store.put(token, data);
        return token;
    }

    public byte[] download(String token) {
        return store.get(token);
    }

    public void delete(String token) {
        store.remove(token);
    }
}

// Message broker
class MessageBroker {
    private final Map<String, Queue<Map<String, Object>>> queues = new ConcurrentHashMap<>();

    public void publish(String queue, Map<String, Object> message) {
        queues.computeIfAbsent(queue, k -> new ConcurrentLinkedQueue<>()).offer(message);
    }

    public Map<String, Object> consume(String queue) {
        Queue<Map<String, Object>> q = queues.get(queue);
        return q != null ? q.poll() : null;
    }
}

// Claim Check Producer
class ClaimCheckProducer {
    private final MessageBroker broker;
    private final BlobStorage storage;

    public ClaimCheckProducer(MessageBroker broker, BlobStorage storage) {
        this.broker = broker; this.storage = storage;
    }

    public String publishLargeMessage(String queue, String payloadJson, Map<String, Object> metadata) {
        byte[] data = payloadJson.getBytes();
        String token = storage.upload(data, "application/json");

        Map<String, Object> message = new HashMap<>();
        message.put("claimCheck", token);
        message.put("metadata", metadata);
        message.put("payloadSize", data.length);
        message.put("timestamp", new Date().toInstant().toString());

        broker.publish(queue, message);
        System.out.println("Published claim check: " + token + " (" + data.length + " bytes)");
        return token;
    }
}

// Claim Check Consumer
class ClaimCheckConsumer {
    private final MessageBroker broker;
    private final BlobStorage storage;

    public ClaimCheckConsumer(MessageBroker broker, BlobStorage storage) {
        this.broker = broker; this.storage = storage;
    }

    public String processNext(String queue) {
        Map<String, Object> message = broker.consume(queue);
        if (message == null) return null;

        String token = (String) message.get("claimCheck");
        byte[] data = storage.download(token);

        if (data == null) {
            System.err.println("ERROR: Claim check not found: " + token);
            return null;
        }

        String payload = new String(data);
        System.out.println("Retrieved payload (" + message.get("payloadSize") + " bytes)");
        storage.delete(token);
        System.out.println("Deleted claim check: " + token);
        return payload;
    }
}

// Usage
BlobStorage storage = new InMemoryBlobStorage();
MessageBroker broker = new MessageBroker();
ClaimCheckProducer producer = new ClaimCheckProducer(broker, storage);
ClaimCheckConsumer consumer = new ClaimCheckConsumer(broker, storage);

String largePayload = "{\"report_id\":\"RPT-001\",\"data\":\"" + "x".repeat(10000) + "\"}";
producer.publishLargeMessage("reports", largePayload, Map.of("priority", "high"));
consumer.processNext("reports");

JavaScript

const crypto = require('crypto');

// Blob storage
class InMemoryBlobStorage {
  constructor() {
    this.store = new Map();
  }

  upload(data, contentType) {
    const token = crypto.randomUUID();
    this.store.set(token, { data, contentType, createdAt: new Date() });
    return token;
  }

  download(token) {
    const entry = this.store.get(token);
    return entry ? entry.data : null;
  }

  delete(token) {
    this.store.delete(token);
  }
}

// Message broker
class MessageBroker {
  constructor() {
    this.queues = new Map();
  }

  publish(queue, message) {
    if (!this.queues.has(queue)) this.queues.set(queue, []);
    this.queues.get(queue).push(message);
  }

  consume(queue) {
    const messages = this.queues.get(queue);
    return messages && messages.length > 0 ? messages.shift() : null;
  }
}

// Claim Check Producer
class ClaimCheckProducer {
  constructor(broker, storage) {
    this.broker = broker;
    this.storage = storage;
  }

  publishLargeMessage(queue, payload, metadata = {}) {
    const payloadBytes = Buffer.from(JSON.stringify(payload));
    const token = this.storage.upload(payloadBytes, 'application/json');

    const message = {
      claimCheck: token,
      metadata,
      payloadSize: payloadBytes.length,
      timestamp: new Date().toISOString()
    };

    this.broker.publish(queue, message);
    console.log(`Published claim check: ${token} (${payloadBytes.length} bytes)`);
    return token;
  }
}

// Claim Check Consumer
class ClaimCheckConsumer {
  constructor(broker, storage) {
    this.broker = broker;
    this.storage = storage;
  }

  processNext(queue) {
    const message = this.broker.consume(queue);
    if (!message) return null;

    const payloadBytes = this.storage.download(message.claimCheck);
    if (!payloadBytes) {
      console.error(`ERROR: Claim check not found: ${message.claimCheck}`);
      return null;
    }

    const payload = JSON.parse(payloadBytes.toString());
    console.log(`Retrieved payload (${message.payloadSize} bytes)`);

    this.storage.delete(message.claimCheck);
    console.log(`Deleted claim check: ${message.claimCheck}`);

    return payload;
  }
}

// Usage
const storage = new InMemoryBlobStorage();
const broker = new MessageBroker();
const producer = new ClaimCheckProducer(broker, storage);
const consumer = new ClaimCheckConsumer(broker, storage);

const largePayload = {
  reportId: 'RPT-001',
  records: Array.from({ length: 1000 }, (_, i) => ({ id: i, data: 'x'.repeat(1000) }))
};

producer.publishLargeMessage('reports', largePayload, { priority: 'high' });
consumer.processNext('reports');

Explanation

The pattern splits each message into two channels: the envelope travels on the broker, the data travels on storage.

1. Store: the producer uploads the payload to object storage and gets back an identifier — a key, a URL or an opaque token.
2. Reference: the producer publishes a small message holding that reference plus metadata (size, content type, checksum).
3. Retrieve: the consumer reads the reference and downloads the payload directly from storage.
4. Cleanup: the stored object is deleted once nobody needs it.

Because the payload and the message live in two systems, you can’t update both atomically. The classic failure modes are a message that arrives before the payload is readable, the same message delivered twice — at-least-once delivery is the norm — and a payload that’s already gone when the consumer shows up. Design around all three: publish the token only after the upload succeeds, retry missing references briefly before giving up, and make processing idempotent by pairing this pattern with the Idempotent Consumer Pattern. A checksum inside the message lets the consumer verify the download before trusting it.

• Companion code: the claim-check-pattern companion repo packages these examples as runnable Python, JavaScript and Java files.

Cleanup Strategies

The hardest operational question is who deletes the payload, and when. Four strategies cover most systems:

Strategy	How it works	Best for	Watch out for
Consumer deletes	Consumer removes the object right after processing	Single consumer, short-lived data	Payloads orphan if the consumer crashes between processing and deleting
TTL / lifecycle rules	Storage auto-expires objects after N hours or days	Any workload; zero code	Too short a TTL leaves consumers holding dead references
Reference counting	Each consumer decrements a counter; the last one deletes	Fan-out to a known set of consumers	The counter needs atomic updates and its own failure handling
Sweeper job	A scheduled worker deletes objects past an age threshold	Safety net alongside another strategy	One more moving part to deploy and monitor

Most production setups combine two: the consumer deletes on the happy path and a TTL lifecycle rule (S3 lifecycle, Azure Blob lifecycle management) reaps whatever a crash or a poison message left behind. Never rely on consumer-deletes alone — a consumer that dies between processing and deletion leaks storage silently.

Security and Integrity

A claim check works as a capability: whoever holds the token can read the payload. Treat it accordingly.

• Encrypt payloads at rest (SSE-S3, SSE-KMS or the equivalent) and require HTTPS for retrieval.
• Prefer short-lived pre-signed URLs over bucket credentials embedded in the message.
• Make tokens unguessable — uuid4 or random keys are fine; sequential IDs let anyone enumerate your payloads.
• Send a SHA-256 hash of the payload in the message so the consumer can detect a truncated or swapped download.
• Scope storage permissions tightly: producers write, consumers read and delete, all on the claim-check prefix only.

Variants

The mechanics are the same everywhere; what changes is where the payload lives and how consumers reach it.

Variant	Storage	Use Case
S3 Claim Check	Amazon S3 / pre-signed URLs	Cross-region, durable, cost-effective
Shared Volume	NFS / EFS / SMB	On-premise, low-latency, same network
Database BLOB	PostgreSQL BYTEA / MySQL BLOB	When transactions with metadata are needed
CDN	CloudFront / Cloudflare	Public content that consumers download directly
Streaming	S3 + byte-range requests	Video/audio where consumers seek to positions

Pick the storage the way you’d pick it for any other large-file problem — durability, latency to consumers, cost — because the messaging side doesn’t change.

What Works

• Use pre-signed URLs for security. Instead of exposing storage directly, the message carries a time-limited URL.
• Expire payloads automatically. Storage lifecycle rules delete old claim checks without a line of cleanup code.
• Send a checksum in the message. A payload hash lets the consumer catch a truncated or swapped download before processing it.
• Plan for missing payloads. If a claim check points nowhere, retry briefly, then log and dead-letter instead of crashing the loop.
• Compress before storing. Gzip or Brotli cuts both the storage bill and the download time on big text payloads.

Common Mistakes

• Never cleaning up. Without deletion or a TTL, claim checks pile up and the storage bill grows quietly month after month.
• Storing sensitive data unencrypted. Encrypt at rest and fetch over HTTPS; an unguessable token isn’t a substitute.
• Issuing URLs that never expire. A leaked pre-signed URL with no expiry is a permanent read key.
• Assuming the payload is there. Objects get deleted early by bad lifecycle rules and lost by custom stores — the consumer must handle missing references.
• Downloading synchronously in the consumer loop. Fetch asynchronously or hand off to a worker so one slow 50MB download doesn’t stall the whole queue.

Real-World Examples

AWS SQS + S3

SQS rejects anything over 256KB, and AWS’s own answer is the claim check: the SQS Extended Client Library stores the payload in S3 and sends the object pointer in the message, handling payloads up to 2GB. You can do the same with plain SDK calls — the library just hides the plumbing.

Azure Service Bus + Blob Storage

Azure Service Bus accepts up to 256KB on Standard and up to 100MB on Premium, but each entity still defaults to a 1MB cap that must be raised explicitly. Microsoft’s architecture center documents claim check as the standard workaround: upload to Blob Storage and send the blob URI in the message.

Kafka + Object Storage

Kafka’s default message.max.bytes sits around 1MB, and raising it costs broker memory and replication bandwidth. Teams producing large payloads — reports, media, batch exports — push them to S3 or HDFS and keep the actual Kafka record under a few KB.

Further Reading

• Claim Check pattern — Azure Architecture Center: Microsoft’s version, including the Service Bus workflow and token-security guidance.
• Managing large Amazon SQS messages: the official Extended Client Library docs for payloads up to 2GB.
• Enterprise Integration Patterns: Hohpe and Woolf’s catalog, where the pattern was first documented.