Bash Loop Over Files

Overview

Looping over files is one of the most common Bash operations, yet it is frequently done incorrectly. Filenames with spaces, newlines, or glob characters (*, ?) break naive loops. This approach shows how to safe, portable patterns for iterating files, filtering by extension, recursing into subdirectories, and processing results.

When to Use

• Running the same command on many files (convert, analyze, move)
• Finding files matching a pattern and processing them in order
• Bulk renaming, permission changes, or validation
• Generating reports from a directory of input files
• Replacing text across multiple files

When NOT to Use

• Processing millions of files — argument list length limits (ARG_MAX) will fail
• Complex filtering that is easier in find with -exec or xargs
• Operations requiring cross-file state — use a proper scripting language (Python, Perl)
• Tasks that need error recovery per file — set -e with loops is tricky

Step-by-Step Implementation

Basic Safe Loop with Glob

#!/bin/bash
set -euo pipefail

# CORRECT: Always quote the variable
txt_count=0
for file in *.txt; do
    # Handle the no-match case (glob leaves literal '*.txt')
    [ -e "$file" ] || continue
    echo "Processing: $file"
    ((txt_count++))
done
echo "Total .txt files: $txt_count"

Recurse with find

#!/bin/bash
set -euo pipefail

# Process all .py files under src/, safely handling spaces
while IFS= read -r -d '' file; do
    echo "Linting: $file"
    pylint "$file"
done < <(find src/ -type f -name '*.py' -print0)

# One-liner version with xargs (no loop needed)
find src/ -type f -name '*.py' -print0 | xargs -0 pylint

# Process with a limit (safer for huge directories)
find src/ -maxdepth 2 -type f -name '*.py' -print0 | \
    xargs -0 -n 10 -P 4 pylint

Filter and Sort

#!/bin/bash

# Numeric sort on filenames like report_001.txt, report_002.txt
for file in $(ls -1 report_*.txt | sort -t_ -k2 -n); do
    echo "Processing in order: $file"
done

# Safer alternative using array + glob
files=(report_*.txt)
IFS=$'\n' sorted=($(sort -t_ -k2 -n <<< "${files[*]}")); unset IFS
for file in "${sorted[@]}"; do
    echo "Ordered: $file"
done

Process Files with Spaces and Special Characters

#!/bin/bash
set -euo pipefail

# Handle filenames with spaces, newlines, and globs
srcdir="/data/uploads"

# Approach 1: read with find -print0
while IFS= read -r -d '' filepath; do
    filename=$(basename "$filepath")
    echo "File: $filename"
done < <(find "$srcdir" -type f -print0)

# Approach 2: shopt nullglob + quoted expansion
shopt -s nullglob
targets=("$srcdir"/*)
shopt -u nullglob

for filepath in "${targets[@]}"; do
    [ -f "$filepath" ] || continue
    echo "Found: $(basename "$filepath")"
done

Bulk Operations

#!/bin/bash
set -euo pipefail

# Rename .jpeg to .jpg
for file in *.jpeg; do
    [ -e "$file" ] || continue
    mv -- "$file" "${file%.jpeg}.jpg"
done

# Convert all HEIC images to JPEG
for file in *.heic; do
    [ -e "$file" ] || continue
    base="${file%.heic}"
    heif-convert "$file" "$base.jpg"
done

# Validate all JSON files
error_count=0
for file in *.json; do
    [ -e "$file" ] || continue
    if ! jq empty "$file" 2>/dev/null; then
        echo "ERROR: Invalid JSON in $file" >&2
        ((error_count++))
    fi
done
[ "$error_count" -eq 0 ] || exit 1

What Works

• Always quote file variables. "$file" prevents word splitting on spaces and interpretation of glob characters.
• Use find -print0 | while read -r -d '' for recursive or complex filtering. It is the only portable way to handle all valid filenames.
• Enable nullglob when using globs in loops. Otherwise *.txt with no matches iterates once with the literal string *.txt.
• Use -- before filenames in commands. mv -- "$file" "$dest" prevents filenames starting with - from being interpreted as options.
• Check [ -e "$file" ] at loop start. Handles both nullglob disabled and empty directory cases.

Common Mistakes

• for file in $(ls *.txt) — never do this. ls output is not parseable; spaces and newlines in filenames break the loop.
• Unquoted variables: mv $file $dest fails on My Document.txt because it splits into two arguments.
• Forgetting nullglob: The loop body runs once with *.txt as the filename when no matches exist.
• Using cat to feed a single file to a program: cat "$file" | grep pattern is a useless use of cat. Use grep pattern "$file".
• Not handling the no-match case: An empty directory with a naive loop can produce unexpected behavior or errors.

Additional Best Practices

• For a deeper guide, see Bash Parallel Execution.

1. Use mapfile for reading file lists into arrays. It is faster than a while read loop for large lists and preserves special characters:

#!/bin/bash
# Read all .py files into an array safely
mapfile -d '' -t pyfiles < <(find src/ -type f -name '*.py' -print0)

echo "Found ${#pyfiles[@]} Python files"
for file in "${pyfiles[@]}"; do
    echo "  $file"
done

2. Use xargs for simple one-command-per-file operations. It handles batching and parallelism automatically:

#!/bin/bash
# Compress all .log files in parallel (4 at a time)
find /var/log -type f -name '*.log' -print0 | xargs -0 -P 4 -I{} gzip "{}"

# Run eslint on all .js files, 20 at a time
find src/ -type f -name '*.js' -print0 | xargs -0 -n 20 eslint

3. Set IFS correctly when parsing command output. The default IFS includes spaces, which breaks filenames. Always use IFS= with read:

#!/bin/bash
# Bad: IFS includes space, breaks on "My File.txt"
# echo "My File.txt" | while read file; do echo "$file"; done
# Output: "My" and "File.txt" on separate lines

# Good: IFS= prevents leading/trailing whitespace trimming
echo "My File.txt" | while IFS= read -r file; do echo "$file"; done
# Output: "My File.txt"

Additional Common Mistakes

1. Using for file in $(find ...) instead of while read. The for loop splits on spaces, breaking filenames with spaces. Always pipe find -print0 into while IFS= read -r -d '':

# Bad: breaks on spaces
# for file in $(find . -name "*.txt"); do echo "$file"; done

# Good: handles all filenames
while IFS= read -r -d '' file; do
    echo "$file"
done < <(find . -name "*.txt" -print0)

2. Not using set -euo pipefail in scripts. Without set -e, errors in the loop body are silently ignored. Without set -u, undefined variables expand to empty strings. Without pipefail, piped commands that fail are masked:

#!/bin/bash
# Bad: errors are silently ignored
# for file in *.txt; do
#     process "$file"
# done

# Good: abort on error, undefined variable, or pipe failure
set -euo pipefail
for file in *.txt; do
    [ -e "$file" ] || continue
    process "$file"
done

3. Modifying files while iterating. Adding, removing, or renaming files during a glob loop can cause skipped or duplicate processing. Collect the file list first, then iterate:

#!/bin/bash
shopt -s nullglob
files=(*.tmp)
shopt -u nullglob

for file in "${files[@]}"; do
    # Safe: we already captured the list
    rm -- "$file"
done