Bash Parallel Execution

Overview

Modern machines ship with plenty of CPU cores, yet most shell scripts run one command at a time. Feeding the same task to several workers (fan-out) then collecting results (fan-in) can cut batch processing time dramatically, and Bash gives you three ways to do it without leaving the terminal: xargs -P, GNU parallel, and background jobs with wait. This recipe covers all three, plus how to cap concurrency and catch failures. For production-grade job control (joblogs, resume, retries with backoff), see Parallel Job Control with Bash.

When to Use

• Processing thousands of files with a CPU-bound tool (image conversion, compression)
• Running tests across many directories or configurations
• Bulk API calls where the remote service tolerates concurrency
• Downloading or encoding many files at once

Skip parallelism when the bottleneck is a single disk, when the remote API rate-limits you, when tasks depend on each other’s output (use a DAG tool like Make), or when you need results in input order; workers finish whenever they finish.

Solution

xargs (POSIX, no extra dependencies)

xargs -P is the lowest-common-denominator option: it’s everywhere, including minimal Docker images.

#!/bin/bash
set -euo pipefail

# Process 4 files at a time; -print0/-0 handles spaces in filenames
find images/ -name '*.png' -print0 | \
    xargs -0 -n 1 -P 4 convert '{}' '{}.jpg'

# Match concurrency to CPU count
find images/ -name '*.png' -print0 | \
    xargs -0 -n 1 -P "$(nproc)" convert '{}' '{}.jpg'

# One command per file (-I implies one input per call); exits 123 on child failure
find data/ -name '*.json' -print0 | \
    xargs -0 -P 4 -I {} sh -c 'jq empty "$1" || echo "FAIL: $1"' _ {}

# Fan out to multiple hosts
printf '%s\n' host1 host2 host3 | \
    xargs -n 1 -P 3 -I {} rsync -avz ./deploy/ {}:/var/app/

GNU Parallel (most capable)

GNU parallel adds progress bars, output grouping, tagged results, and remote execution:

#!/bin/bash
set -euo pipefail

# Basic run with progress bar; {.} strips the extension
find images/ -name '*.png' | parallel --bar convert '{}' '{.}.jpg'

# Cap concurrency, preserve input order in the output
find logs/ -name '*.log' | parallel -j 8 --keep-order gzip '{}'

# Per-job result files under ./logs/ instead of interleaved stdout
find data/ -name '*.csv' | \
    parallel -j 4 --results 'logs/{=s:.*/::;s:\.csv::=}' \
        awk -F, '{print $2}' {}

# Parallel SSH across a fleet, tagged per host
parallel -j 10 --tag ssh {} uptime ::: server1 server2 server3

For joblogs and resuming failed runs (--joblog, --resume-failed), see the job control recipe.

Bash background jobs (zero dependencies)

#!/bin/bash
set -uo pipefail   # no -e: it can't catch background failures anyway

MAX_JOBS=4
pids=()

for file in *.mp4; do
    [ -e "$file" ] || continue
    # Bash 4.3+: wait -n blocks until any job finishes — no sleep polling
    while [ "$(jobs -rp | wc -l)" -ge "$MAX_JOBS" ]; do
        wait -n 2>/dev/null || sleep 0.1
    done
    ffmpeg -i "$file" "${file%.mp4}.webm" &
    pids+=($!)
done

# Collect each job's exit code — wait for them by PID, not bare `wait`
failed=0
for pid in "${pids[@]}"; do
    if ! wait "$pid"; then
        echo "Job $pid failed" >&2
        ((failed++))
    fi
done
[ "$failed" -eq 0 ] || exit 1

Two details matter here: pids+=($!) records each job as it launches, and wait "$pid" (per Bash job control docs) returns that job’s real exit code. A bare wait first would reap everything and leave jobs -p empty. Your failure check would never see a thing.

Semaphore pattern for rate-limited APIs

GNU parallel ships sem, a counting semaphore that caps concurrent calls:

#!/bin/bash
set -euo pipefail

API_LIMIT=10  # max concurrent calls

while read -r id; do
    sem --id api_calls -j "$API_LIMIT" \
        curl -sf "https://api.example.com/items/$id" -o "results/$id.json" &
done < ids.txt

wait
sem --id api_calls --wait  # drain any stragglers

Useful flags for flaky or uneven work

#!/bin/bash
# Retry each failed job up to 3 times
parallel -j 4 --retries 3 wget -q :::: urls.txt

# Kill any job that runs longer than 30s
parallel -j 8 --timeout 30 curl -s :::: urls.txt

# Even out the work when input sizes vary wildly
find . -name '*.log' -exec du -b {} + | sort -n | \
    parallel --round-robin -j 4 gzip {2}

# Progress with xargs (it has none) via pv
xargs -P4 -I{} gzip {} < filelist.txt   # blind
pv -l < filelist.txt | xargs -P4 -I{} gzip {}

# Pin each job to a specific core for CPU-bound work
parallel -j "$(nproc)" 'taskset -c %{} gzip {}' ::: *.log

For plain shell functions, wrap the call in timeout 30 instead of relying on --timeout — it works with xargs and background jobs too.

Explanation

Every approach follows the same shape:

The differences are in the plumbing, not the shape. xargs is a dumb fan-out: no progress, no ordering, exit code 123 as its only failure signal. GNU parallel is the full toolkit: ordering, grouping, tagged output, remote hosts. Raw jobs/wait is for when neither tool is installed or you need custom logic between launches, like the semaphore above.

What Works

• Set -P/-j explicitly. xargs defaults to sequential; GNU parallel defaults to core count, which can still swamp I/O-bound work.
• Use -print0 | xargs -0. Filenames with spaces or newlines break naive whitespace splitting.
• Prefer wait -n over sleep polling for slot management; it wakes the moment a job exits instead of every 100ms.
• Redirect per-job output to files. Interleaved stdout is undebuggable; --results (parallel) or a redirect per job keeps each log intact.
• Use --halt soon,fail=1 for fail-fast. When one bad input should stop the batch, don’t let the other 500 jobs burn CPU first: find tests/ -name '*.sh' | parallel -j 8 --halt soon,fail=1 bash {}.
• Trap SIGINT and kill the children. Ctrl+C on the parent leaves background jobs running; trap 'jobs -p | xargs -r kill' SIGINT SIGTERM cleans up.

Common Mistakes

1. Assuming set -e catches background failures. It doesn’t: a failing & job won’t exit the script unless you wait on its PID and check. Collect pids+=($!) and verify each one.
2. wait then jobs -p. A bare wait reaps every job, so jobs -p afterwards returns nothing and your exit-code loop silently does nothing. Save PIDs when you launch.
3. Interleaved output. Concurrent writes to stdout produce garbled lines; group with --group/--keep-order or redirect per job.
4. Subshells that don’t share state. ((counter++)) & updates a copy; the parent’s counter stays 0. Use result files, flock on a shared file, or GNU parallel’s result handling.
5. Quoting bugs in xargs -I {}. {} inside single quotes won’t expand where you expect; pass it as a positional arg (sh -c '... "$1"' _ {}) so filenames can’t inject shell.
6. Ignoring xargs exit 123. It means “at least one child failed”; check $? or the failures pass silently.

Troubleshooting

• xargs exits 123 with no error shown. One or more children failed; their stderr was probably interleaved into stdout. Re-run with per-job logs to see which input broke.
• GNU parallel prints a citation notice in CI. It’s a first-run nag, not an error. Run parallel --citation once interactively, or pass --will-cite where you call it.
• Jobs keep running after Ctrl+C. The parent died but children didn’t. Add the trap cleanup from What Works, or run under timeout.
• Everything runs sequentially anyway. Check that -P/-j is actually set (xargs defaults to 1), and that input isn’t a single line: xargs batches whitespace-separated items, so cat file | xargs -P4 on one line feeds it all to one job.
• Machine thrashes with parallel jobs. Your limit is above what RAM or I/O can take. Drop -j below core count for I/O-bound work, or use --memfree (parallel) to hold back jobs under memory pressure.