Background Jobs: Task Queues, Cron & Workers

Overview

When a user uploads a file, places an order, or signs up, the HTTP response should come back in milliseconds. The PDF render, the welcome email, or the third-party sync can happen a few seconds later in a separate process — that’s what a background job is for. Below are working task-queue and cron setups in Python (Celery + Redis), JavaScript (BullMQ + Redis), and Java (Spring @Scheduled), plus the retry, dead-letter, and monitoring patterns you need to run them in production.

When to Use

Reach for a background job when:

• An email or SMS can leave a few seconds after the request finishes. See Email Templates MJML for the content side.
• An export, report, or PDF takes more than a second to build. See Generate PDFs for the document part.
• An uploaded image, video, or document needs resizing, transcoding, or scanning. See Image Optimization for media processing.
• Data has to sync with an external API on a fixed schedule. See Call REST API for the client side.
• Nightly aggregation or cleanup runs on a clock rather than on user events. See Scheduled Jobs for serverless scheduled tasks.

Don’t push work to a queue when the user is actively waiting on the result — form validation, auth checks, and reads belong in the request path. Tiny one-off tasks on serverless (under ~100 ms) are also usually cheaper as direct invocations than as a queue hop.

Solution

Python (Celery + Redis)

from celery import Celery
from celery.schedules import crontab

app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/0")

@app.task(bind=True, max_retries=3)
def send_email(self, to, subject, body):
    try:
        # Real implementation goes through SMTP or an API like SendGrid
        print(f"Sending email to {to}")
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

@app.task
def generate_report(user_id):
    print(f"Generating report for user {user_id}")
    return f"/reports/{user_id}.pdf"

@app.task
def cleanup_old_logs():
    print("Deleting log files older than 30 days")

# Periodic tasks run through celery beat
app.conf.beat_schedule = {
    "daily-cleanup": {
        "task": "tasks.cleanup_old_logs",
        "schedule": crontab(hour=2, minute=0),
    },
}

# From the web app:
send_email.delay("alice@example.com", "Welcome", "Hello!")

JavaScript (BullMQ + Redis)

const { Queue, Worker } = require("bullmq");
const IORedis = require("ioredis");

// BullMQ workers require maxRetriesPerRequest: null
const connection = new IORedis({ host: "localhost", port: 6379, maxRetriesPerRequest: null });

const emailQueue = new Queue("emails", { connection });

const worker = new Worker("emails", async (job) => {
  const { to, subject, body } = job.data;
  console.log(`Sending email to ${to}`);
  return { sent: true };
}, { connection });

async function main() {
  // From an API route:
  await emailQueue.add("send-email", {
    to: "alice@example.com", subject: "Welcome", body: "Hello!",
  }, {
    attempts: 3,
    backoff: { type: "exponential", delay: 1000 },
  });

  // Recurring job: cleanup every day at 02:00
  const cronQueue = new Queue("cron", { connection });
  await cronQueue.add("cleanup", {}, { repeat: { pattern: "0 2 * * *" } });
}

main().catch(console.error);

Java (Spring @Scheduled)

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Service;
import java.util.concurrent.CompletableFuture;

@Service
public class JobService {

    // Requires @EnableScheduling on a @Configuration class
    @Scheduled(cron = "0 0 2 * * ?")   // every day at 02:00
    public void cleanupOldLogs() {
        System.out.println("Running nightly cleanup");
    }

    @Scheduled(fixedRate = 300_000)    // every 5 minutes
    public void syncExternalData() {
        System.out.println("Syncing with external API");
    }

    // Fire-and-forget work inside a request goes through a pool
    public void runAsync(Runnable task, ThreadPoolTaskExecutor executor) {
        CompletableFuture.runAsync(task, executor);
    }
}

Explanation

Three pieces make up any background-job setup:

1. A producer (your API or web app) enqueues a job when something happens — a signup, an upload, a scheduled tick.
2. A broker (Redis, RabbitMQ, Amazon SQS) stores the job until a worker picks it up. Redis with AOF persistence is enough for most apps; RabbitMQ or SQS earns its operational cost when losing a job means losing money.
3. A consumer (the worker process) polls the broker and runs the job. Workers scale horizontally and can live on different machines than the API.

Cron jobs are the same pattern with a clock as the producer: Celery Beat, BullMQ’s repeatable jobs, and Spring’s @Scheduled all enqueue work on a schedule instead of on an event.

One thing that surprises people: delivery is at-least-once, not exactly-once. A worker can finish the work and crash before acknowledging the job, so the broker hands it to another worker and the side effect runs twice. Celery only removes the job after the task finishes when task_acks_late=True; BullMQ re-queues a stalled job once its lock expires; SQS does the same with its visibility timeout. Design every job so running it twice is harmless — check a marker, carry an idempotency key, or make the write an upsert.

Variants

Tool	Language	Persistence	Scheduling	Best For
Celery + Redis	Python	Redis (or RabbitMQ)	Celery Beat	Full-featured task queues
BullMQ	JavaScript	Redis	Built-in cron	Node.js, TypeScript projects
Sidekiq	Ruby	Redis	sidekiq-cron	Ruby on Rails
Hangfire	C#	SQL Server/Redis	Built-in	.NET ecosystem
Spring @Scheduled	Java	N/A (in-process)	Cron expressions	Simple scheduled tasks
AWS Lambda + EventBridge	Any	N/A (serverless)	EventBridge rules	Cloud-native, pay-per-use

What Works

• Make jobs idempotent. Delivery is at-least-once in every queue system worth using, so a job can run twice. A unique job ID plus an upsert — or a dedupe key like email:42:welcome written with SET NX — keeps the second run harmless.
• Retry transient failures with backoff. A timeout to your SMTP provider or a flaky API deserves 3–5 retries with exponential backoff and jitter, not an immediate trip to the dead-letter queue.
• Log the job context. Job ID, queue name, attempt number, and user ID in every line turns “something failed” into a queryable event.
• Split queues by priority. Payment processing on high, email on default, nightly cleanup on low — a backlog of cheap jobs shouldn’t delay expensive ones.
• Watch the dead-letter queue. A job that exhausts its retries is either a bug or a poison message. Alert on DLQ growth instead of discovering it weeks later.

Common Mistakes

• Running heavy work in the web process. A 100-page PDF inside the request path means a timeout for that user and CPU contention for everyone else.
• Skipping retry and dead-letter config. A Redis restart without persistence configured can drop every pending job; an unhandled exception can drop the same job over and over.
• Assuming exact cron timing. Cron means “at or after” the scheduled time — a busy worker can delay a 02:00 job by minutes. Don’t chain jobs that assume the previous one ran to the second.
• Not planning for mid-job crashes. Without acknowledgments or visibility timeouts, a dead worker takes its in-flight job with it.
• Flooding the queue. Enqueuing 100,000 jobs at once starves real-time work. Batch the enqueue or rate-limit it.

Advanced Patterns

Runnable versions of these examples live in the companion repository.

Docker Compose for Local Development

# docker-compose.yml
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  worker:
    build: .
    command: celery -A tasks worker --loglevel=info --concurrency=4
    depends_on: [redis]
    environment:
      - REDIS_URL=redis://redis:6379/0

  beat:
    build: .
    command: celery -A tasks beat --loglevel=info
    depends_on: [redis, worker]
    environment:
      - REDIS_URL=redis://redis:6379/0

  api:
    build: .
    ports: ["8000:8000"]
    depends_on: [redis]
    environment:
      - REDIS_URL=redis://redis:6379/0

Priority Queues in Celery

from celery import Celery
from kombu import Queue

app = Celery("tasks", broker="redis://localhost:6379/0")

# Route work into separate queues by priority
app.conf.task_queues = (
    Queue("high", routing_key="high.#"),
    Queue("default", routing_key="default.#"),
    Queue("low", routing_key="low.#"),
)

app.conf.task_routes = {
    "tasks.process_payment": {"queue": "high"},
    "tasks.send_email": {"queue": "default"},
    "tasks.cleanup_logs": {"queue": "low"},
}

# A worker consumes queues in order — high first
# celery -A tasks worker -Q high,default,low

With a Redis broker, priority is approximated by consuming queues in order. RabbitMQ supports real message priority through the x-max-priority queue argument.

Dead-Letter Queues in BullMQ

const { Queue, Worker } = require("bullmq");

const emailQueue = new Queue("emails", { connection });
const dlqQueue = new Queue("dlq", { connection });

const worker = new Worker("emails", async (job) => {
  const { to, subject, body } = job.data;
  if (to.includes("fail")) throw new Error("Simulated failure");
  console.log(`Email sent to ${to}`);
}, {
  connection,
  attempts: 3,
  backoff: { type: "exponential", delay: 2000 },
});

// "failed" fires on every attempt — only move to the DLQ on the last one
worker.on("failed", async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    console.error(`Job ${job.id} exhausted retries: ${err.message}`);
    await dlqQueue.add("failed-email", job.data, { removeOnComplete: true });
  }
});

Tracking Job Progress

@app.task(bind=True)
def long_running_task(self, items):
    total = len(items)
    for i, item in enumerate(items):
        process(item)
        self.update_state(
            state="PROGRESS",
            meta={"current": i + 1, "total": total, "percent": (i + 1) / total * 100}
        )
    return {"status": "complete", "processed": total}

# The API stores the task_id returned by .delay() and polls with it:
result = app.AsyncResult(task_id)
print(result.info)  # {'current': 50, 'total': 100, 'percent': 50.0}

Performance Tips

1. Batch small jobs. Processing 1,000 items one job at a time costs 1,000 round-trips to the broker. Batch them:

@app.task
def process_batch(item_ids):
    # Django ORM shown; any store works
    items = Item.objects.filter(id__in=item_ids)
    for item in items.iterator():
        process(item)

# ids comes from whatever set you need to process
for i in range(0, len(ids), 100):
    process_batch.delay(ids[i:i + 100])

2. Cap prefetch. Without limits, one worker grabs every job and the rest idle:

# Each worker fetches only 1 job at a time
celery -A tasks worker --prefetch-multiplier=1 --concurrency=4

3. Give latency-sensitive jobs their own queue. A 5-minute video encode shouldn’t sit ahead of a password reset:

celery -A tasks worker -Q realtime --concurrency=8
celery -A tasks worker -Q batch --concurrency=2

See Also

• Celery documentation — task routing, retries, and Beat scheduling
• BullMQ documentation — repeatable jobs, rate limiting, and flows
• Spring scheduling reference — @Scheduled and @EnableScheduling
• Amazon EventBridge Scheduler — serverless scheduled invocations
• RabbitMQ tutorials — work queues and acknowledgment semantics