Parallel Job Execution with Bash

Overview

Modern servers have multiple cores, but a naive shell script runs one command at a time. Parallel job execution lets you process many files, URLs, or tasks at once, cutting runtime from hours to minutes. Bash gives you several tools for this: background jobs, xargs -P, and GNU parallel. Each option balances control, portability, and ease of use.

When to Use

Use this resource when:

• You need to process many files or records in a batch.
• A sequential loop is too slow for your workflow.
• You want to control the maximum number of concurrent jobs.
• You need to collect exit codes from every child process.

Solution

Bash parallel job execution

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS="${1:-4}"
INPUT_FILE="${2:-jobs.txt}"

# Option 1: xargs with parallel workers
process_task() {
    local task="$1"
    echo "Processing $task"
    sleep "$((RANDOM % 3 + 1))"
    echo "Done $task"
}
export -f process_task

cat "$INPUT_FILE" | xargs -P "$MAX_JOBS" -I {} bash -c 'process_task "{}"'

# Option 2: GNU parallel
# parallel -j "$MAX_JOBS" process_task {} < "$INPUT_FILE"

# Option 3: Background jobs with a semaphore
SEMAPHORE=0
while IFS= read -r task; do
    if [[ $SEMAPHORE -ge $MAX_JOBS ]]; then
        wait -n
        SEMAPHORE=$((SEMAPHORE - 1))
    fi
    process_task "$task" &
    SEMAPHORE=$((SEMAPHORE + 1))
done < "$INPUT_FILE"
wait

Explanation

The script shows three common approaches. xargs -P is portable and available on most systems, but less flexible than GNU parallel. GNU parallel offers better output handling, resuming, and progress display. The background-jobs approach uses wait -n to keep a maximum number of concurrent jobs without external tools. export -f makes the Bash function visible to subprocesses when using xargs -I {} bash -c.

Variants

Approach	Tool	Pros	Cons
xargs	coreutils	Portable, simple	Limited control, messy output
GNU parallel	parallel	capable, resumable, ordered output	Extra dependency
Background jobs	bash builtin	No external deps	Manual bookkeeping, race-prone

What Works

1. Cap concurrency to a tested limit. Too many jobs exhaust CPU, memory, or file descriptors.
2. Make jobs idempotent. A retried job should produce the same result without side effects.
3. Capture and aggregate exit codes. A single failed job should not silently hide among successful ones.
4. Use a temporary directory per job. This prevents file collisions and makes cleanup easy.
5. Log with the job identifier. Prefix output with the task name so you can trace failures.

Common Mistakes

1. Unbounded parallelism. Launching every task in the background at once can crash the shell.
2. Losing exit codes. xargs returns the last exit code by default; use -P with -t or GNU parallel to track each job.
3. Ignoring shell quoting. Filenames with spaces break xargs unless you use -0 or -d.
4. Writing to the same output file. Concurrent writes interleave output; use one file per job or lock the file.
5. No timeout. A stuck job can block the whole batch; add timeout to each command.

Additional Best Practices

• For a deeper guide, see Bash Parallel Execution.

1. Use nproc to auto-detect CPU cores. Instead of hardcoding MAX_JOBS, derive it from the system’s CPU count. For I/O-bound tasks, use 2-4x the core count; for CPU-bound tasks, use 1x:

CPU_CORES=$(nproc)
IO_JOBS=$((CPU_CORES * 4))  # I/O-bound: 4x cores
CPU_JOBS=$CPU_CORES          # CPU-bound: 1x cores

2. Use --bar with GNU parallel for progress display. For long-running batches, a progress bar helps estimate completion time:

parallel --bar --jobs "$MAX_JOBS" process_task {} < "$INPUT_FILE"

3. Set file descriptor limits for high concurrency. Each background job uses file descriptors. For more than 100 concurrent jobs, increase the ulimit:

ulimit -n 4096  # Allow 4096 open file descriptors