StackPractices
intermediate By Mathias Paulenko

Parallel Job Control with Bash

How to control parallel Bash jobs once they're running: collect exit codes, retry failures, enforce timeouts, and resume interrupted batches.

Overview

Getting commands to run in parallel is the easy half — xargs -P, GNU parallel, or a & loop does that in one line (covered in Bash Parallel Execution). The hard half starts once the jobs are running: which ones failed? Which need a retry? What do you rerun after a Ctrl+C halfway through a 500-job batch?

This recipe is about that second half: per-job exit codes, timeouts, retries with backoff, and the GNU parallel joblog that ties it all together.

When to Use

  • A batch where individual jobs can fail and you need to know exactly which ones.
  • Flaky tasks — network calls, rate-limited APIs — that deserve a retry before you give up.
  • Long pipelines that must resume where they stopped instead of starting over.
  • Jobs that can hang and need a hard timeout so one stuck task doesn’t block the batch.

If you just need “run these commands at once,” start with the basic execution recipe and come back here when the batch grows up.

Solution

A complete job-control wrapper: GNU parallel runs the work, writes a joblog per run, and a report tells you what failed.

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS="${1:-4}"
INPUT_FILE="${2:-jobs.txt}"
LOG_DIR="${3:-./parallel-logs}"

mkdir -p "$LOG_DIR"

process_task() {
    local task="$1"
    echo "Processing $task"
    sleep "$((RANDOM % 3 + 1))"
    echo "Done $task"
}
export -f process_task

# Run with a joblog: every job's exit code, duration, and command is recorded
parallel --jobs "$MAX_JOBS" \
    --joblog "$LOG_DIR/joblog.txt" \
    --timeout 300 \
    --halt soon,fail=20% \
    process_task {} \
    < "$INPUT_FILE"

# Summarize: joblog columns are Seq Host Starttime Runtime Send Receive Exitval Signal Command
awk 'NR>1 {codes[$7]++} END {for (c in codes) printf "Exit %s: %d jobs\n", c, codes[c]}' \
    "$LOG_DIR/joblog.txt"

# List what failed so you can rerun just those
awk 'NR>1 && $7!=0 {print $NF}' "$LOG_DIR/joblog.txt" > "$LOG_DIR/failed.txt" || true
if [[ -s "$LOG_DIR/failed.txt" ]]; then
    echo "Failed jobs written to $LOG_DIR/failed.txt"
fi

Explanation

The joblog is the whole trick. GNU parallel appends one row per job — when it started, how long it took, its exit value, and the exact command. That file turns a black-box batch into something you can audit, post-process with awk, and resume.

Three properties make this safe for real batches:

  • Exit codes survive. Each row keeps the child’s exit value, so a failed job can’t hide among 200 successes. xargs -P only tells you “something failed” via exit code 123; the joblog tells you what failed.
  • Timeouts are per job. --timeout 300 kills a single hung task, not the batch. The killed job lands in the joblog with signal 15 so you can tell it apart from a clean failure.
  • --halt soon,fail=20% is your circuit breaker. If more than 20% of jobs fail, parallel stops queuing new work but lets running jobs finish. Tune or drop it for batches where partial failure is expected.
Job control flow. A jobs file feeds a worker pool. Each job runs under a timeout with retries, writes its result to the joblog, and a final aggregation step reports successes and failed jobs for rerun.

Variants

ToolJob control mechanismBest for
GNU parallel--joblog, --resume, --resume-failed, --retriesProduction batches; resumable pipelines
xargs -PPer-job result files (below)Systems where parallel isn’t installed
Background jobs + wait$! PID tracking + wait "$pid"Small batches, zero dependencies
Python multiprocessingReturn codes in pool.map resultsJobs already wrapped in Python

What Works

  1. Write a result file per job. When there’s no joblog, a task_result file per job is the poor-man’s equivalent — and it survives crashes.
  2. Retry with backoff, not loops. Flaky tasks deserve 2–3 attempts with growing delays, then a clean FAILED in the report.
  3. Keep the input file as the source of truth. wc -l jobs.txt vs. result-file count tells you if anything was skipped.
  4. Log with the job identifier. Prefix output with the task name; you’ll thank yourself when debugging one job out of hundreds.
  5. Clean up temp state on exit. trap + mktemp -d keeps result dirs from leaking into the next run.

The GNU Parallel tutorial covers --joblog, --resume and --retries in more depth if you want the full flag reference.

Common Mistakes

  1. Trusting xargs exit code alone. -P returns 123 if any child failed — with zero info about which. Collect per-job status explicitly.
  2. Retrying non-idempotent jobs. If a task has side effects, a retry can double-apply them. Mark which jobs are safe to rerun.
  3. sleep-based semaphore loops. Polling jobs -r in a busy loop wastes CPU; wait -n (Bash 4.3+) blocks until a slot frees.
  4. Timeout as the only failure mode. A job that exits 1 in 100ms isn’t hung — distinguish signals from exit codes in your report.
  5. Resuming without a joblog. --resume needs the joblog from the interrupted run; without it, you’re rerunning everything.

Advanced Solutions

GNU parallel: resume an interrupted batch

--resume reads the joblog and skips jobs that already exited 0. --resume-failed also reruns the ones that failed — the standard “finish the batch” command.

#!/usr/bin/env bash
set -euo pipefail

LOG_DIR="${1:-./parallel-logs}"
INPUT_FILE="${2:-jobs.txt}"

process_task() {
    local task="$1"
    echo "Processing $task"
    sleep "$((RANDOM % 3 + 1))"
}
export -f process_task

# Rerun only what didn't finish cleanly last time
parallel --jobs 4 \
    --joblog "$LOG_DIR/joblog.txt" \
    --resume-failed \
    --timeout 300 \
    process_task {} \
    < "$INPUT_FILE"

xargs: per-job result files

No GNU parallel? Give each job its own result file and aggregate at the end. basename does the filename cleanup that {//} would do in parallel — xargs -I {} doesn’t support {//}.

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS="${1:-4}"
INPUT_FILE="${2:-jobs.txt}"
RESULTS_DIR=$(mktemp -d)
trap 'rm -rf "$RESULTS_DIR"' EXIT

process_with_exit() {
    local task="$1"
    local result_file="$2"
    sleep "$((RANDOM % 3 + 1))"
    if (( RANDOM % 10 == 0 )); then
        echo "FAIL" > "$result_file"
        return 1
    fi
    echo "OK" > "$result_file"
    return 0
}
export -f process_with_exit
export RESULTS_DIR

xargs -P "$MAX_JOBS" -I {} bash -c '
    task="{}"
    process_with_exit "$task" "$RESULTS_DIR/$(basename "$task")_result" || true
' < "$INPUT_FILE"

TOTAL=$(wc -l < "$INPUT_FILE")
SUCCESS=$(grep -l "OK" "$RESULTS_DIR"/*_result 2>/dev/null | wc -l)
echo "Total: $TOTAL, Success: $SUCCESS, Failed: $((TOTAL - SUCCESS))"

Retries with exponential backoff

Wrap the task in a retry function and let parallel fan it out. timeout gives each attempt a hard ceiling; the backoff gives rate-limited services room to breathe.

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS="${1:-4}"
MAX_RETRIES="${2:-3}"
TIMEOUT="${3:-60}"
INPUT_FILE="${4:-jobs.txt}"

process_task() {
    local task="$1"
    # Replace with real work; must succeed on retry
    sleep "$((RANDOM % 3 + 1))"
    (( RANDOM % 3 != 0 ))  # fails ~1 in 3 for demo purposes
}
export -f process_task

run_with_retry() {
    local task="$1"
    local attempt=1

    while (( attempt <= MAX_RETRIES )); do
        if timeout "$TIMEOUT" bash -c 'process_task "$1"' _ "$task"; then
            echo "SUCCESS: $task (attempt $attempt)"
            return 0
        fi
        echo "RETRY: $task (attempt $attempt failed)" >&2
        ((attempt++))
        sleep "$((2 ** (attempt - 1)))"  # 2s, 4s, 8s...
    done

    echo "FAILED: $task after $MAX_RETRIES attempts"
    return 1
}
export -f run_with_retry
export MAX_RETRIES TIMEOUT

parallel --jobs "$MAX_JOBS" run_with_retry {} < "$INPUT_FILE"

The retry loop itself is a small state machine — worth picturing before you tweak it:

Retry loop with exponential backoff. Each attempt runs the job under a timeout. On failure the attempt counter grows; while retries remain, the worker sleeps 2^n seconds and tries again. When retries run out, the job is marked FAILED for a rerun pass.

The timeout command is GNU coreutils — on macOS install coreutils and call it gtimeout.

Python multiprocessing comparison

When the pipeline already lives in Python, multiprocessing.Pool gives you the same control natively — exit codes come back in pool.map results and there’s no quoting to fight. Here each line of jobs.txt is a full shell command:

import multiprocessing
import shutil
import subprocess
import time
from pathlib import Path

# Resolve bash explicitly — on Windows, CreateProcess would otherwise
# find the WSL launcher in System32 before the real bash on PATH
BASH = shutil.which("bash") or shutil.which("sh") or "bash"

def run_task(task: str) -> tuple[str, int, float]:
    start = time.time()
    result = subprocess.run(
        [BASH, "-c", task],
        capture_output=True,
        text=True,
        timeout=300,
    )
    return task, result.returncode, time.time() - start

def main():
    tasks = Path("jobs.txt").read_text().strip().split("\n")
    max_workers = min(4, multiprocessing.cpu_count())

    with multiprocessing.Pool(max_workers) as pool:
        results = pool.map(run_task, tasks)

    succeeded = sum(1 for _, code, _ in results if code == 0)
    print(f"Total: {len(results)}, Success: {succeeded}, Failed: {len(results) - succeeded}")

    for task, code, elapsed in results:
        status = "OK" if code == 0 else f"FAIL (exit {code})"
        print(f"  {task}: {status} ({elapsed:.1f}s)")

if __name__ == "__main__":
    main()

find + xargs: batch file processing

The classic parallel file fan-out — note -print0 | xargs -0 so filenames with spaces don’t break, and every {} is quoted inside the sh -c wrappers:

#!/usr/bin/env bash
set -euo pipefail

MAX_JOBS="${1:-$(nproc)}"
TARGET_DIR="${2:-.}"

# Compress all log files in parallel
find "$TARGET_DIR" -type f -name "*.log" -print0 | \
    xargs -0 -P "$MAX_JOBS" -I {} gzip -9 "{}"

# Generate checksums in parallel
find "$TARGET_DIR" -type f -print0 | \
    xargs -0 -P "$MAX_JOBS" -I {} sh -c 'sha256sum "{}" > "{}.sha256"'

# Resize images in parallel (requires ImageMagick)
find "$TARGET_DIR" -type f \( -name "*.jpg" -o -name "*.png" \) -print0 | \
    xargs -0 -P "$MAX_JOBS" -I {} sh -c 'convert "{}" -resize 50% "{}.thumb"'

Companion code: The bash-parallel-job-execution companion repo has all four job-control scripts as runnable files.

Frequently Asked Questions

How do I rerun only the jobs that failed?

With GNU parallel, keep --joblog on the original run and rerun with --resume-failed. Without it, write a result file per job and feed the missing/failed ones back as a new input file.

How do I collect exit codes from xargs jobs?

xargs -P only returns 123 when something fails — not what failed. Have each job write a result file (OK/FAIL plus the exit code) and aggregate those, or switch to GNU parallel's joblog for a built-in record.

How do I stop a hung job without killing the whole batch?

Give each job its own timeout: GNU parallel's --timeout or wrap the command in timeout 60. The killed job shows up as signal 15 in the joblog, so timeouts are distinguishable from real failures.

What's the difference between xargs and GNU parallel for job control?

xargs fans work out; that's where it stops. GNU parallel adds the control layer: joblog, resume, ordered output, per-job results, remote execution. For anything beyond "run N at once," parallel earns its dependency.

How do I see which jobs are still running?

jobs -l lists background jobs in the current shell, ps aux | grep <command> across the system, and column -t < joblog.txt shows live progress for GNU parallel — rows appear as jobs finish.