Overview

Each Unix tool does one thing well, and pipes compose them: and the shell connects them with pipes. For log analysis, data extraction, and report generation, a one-liner often beats a hundred-line script in another language. Below are the essential tools and how to combine them safely.

When to Use

• Extracting and filtering log lines by pattern, time, or status code
• Transforming CSV or tabular data (sorting, deduplication, aggregation)
• Searching codebases for patterns across thousands of files
• Generating quick reports from structured text output
• Pre-processing data before feeding it to a database or API

When NOT to Use

• Parsing nested or irregular formats (JSON, XML, HTML) — use jq, xq, or a proper parser
• Tasks requiring complex state across lines — awk can do it, but Python is more maintainable
• Multi-step transformations where error handling matters — scripting languages have better debugging
• Unicode edge cases — classic tools are byte-oriented and may mangle multibyte characters

Step-by-Step Implementation

grep — Pattern Matching

# Search recursively, show line numbers, ignore binary files
grep -rn "ERROR" logs/

# Invert match, count occurrences
grep -vc "^#" config.ini

# Multiple patterns with extended regex
grep -E "(ERROR|FATAL|CRITICAL)" app.log

# Context lines: 2 before, 3 after
grep -B 2 -A 3 "Exception" app.log

# Only filenames containing match (useful for batch operations)
grep -rl "TODO" src/

# Perl-compatible regex (PCRE) for lookaheads
grep -P "(?<=user_id=)\d+" access.log

sed — Stream Editing

# Replace first occurrence per line
sed 's/foo/bar/' file.txt

# Replace all occurrences globally
sed 's/foo/bar/g' file.txt

# Replace in-place with backup
sed -i.bak 's/old_domain/new_domain/g' config.conf

# Delete lines matching pattern
sed '/^#/d' config.ini        # Remove comments
sed '/^$/d' file.txt          # Remove empty lines

# Extract specific lines
sed -n '10,20p' file.txt      # Print lines 10-20
sed -n '50,$p' file.txt       # Print from line 50 to end

# Multi-line replacement (append after match)
sed '/pattern/a\\New line after match' file.txt

awk — Field Processing and Aggregation

# Print specific columns (space/tab delimited)
awk '{print $1, $3}' access.log

# Sum a column
awk '{sum += $2} END {print sum}' sales.txt

# Average with count
awk '{sum += $2; count++} END {if (count) print sum/count}' data.txt

# Filter rows by condition
awk '$3 > 100 {print $1, $3}' orders.csv

# Process CSV with custom delimiter
awk -F',' '{print $2, $5}' customers.csv

# Group by and count (like SQL GROUP BY)
awk '{count[$1]++} END {for (k in count) print k, count[k]}' status.log

# Format output with headers
awk 'BEGIN {print "IP", "Requests"} {count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log

cut, sort, uniq — Column Extraction and Deduplication

# Extract columns by position or delimiter
cut -d',' -f1,3,5 data.csv
cut -c1-10 file.txt           # First 10 characters

# Sort numerically, reverse, by specific column
sort -t',' -k3 -n sales.csv   # Sort by 3rd column numerically
sort -u file.txt               # Sort and remove duplicates

# Count unique occurrences
sort file.txt | uniq -c | sort -rn   # Most frequent first

# Show only duplicate or unique lines
sort file.txt | uniq -d       # Only duplicates
sort file.txt | uniq -u       # Only unique lines

tr — Character Translation

# Convert to uppercase
cat file.txt | tr 'a-z' 'A-Z'

# Squeeze repeated characters
tr -s ' ' < file.txt           # Collapse multiple spaces to one

# Delete characters
tr -d '\r' < file.txt          # Remove carriage returns

# Replace line endings
tr '\n' ',' < lines.txt > comma-separated.txt

Complex Pipelines

# Top 10 most frequent error types in a log
awk '$0 ~ /ERROR|FATAL/ {print $5}' app.log | \
    sort | uniq -c | sort -rn | head -10

# Extract unique client IPs with request count, sorted
awk '{print $1}' access.log | sort | uniq -c | sort -rn | \
    awk '{print $2 "," $1}' > ip_counts.csv

# Find slow queries (>1s) and group by table
awk '$NF > 1 {print}' slow_query.log | \
    grep -oP 'FROM \K\w+' | sort | uniq -c | sort -rn

# Convert log timestamps to ISO format and filter a date range
sed -n '/2024-06-01/,/2024-06-07/p' app.log | \
    awk '{gsub(/\//, "-", $1); print $1 "T" $2}'

# Generate a report: status code distribution
awk '{print $9}' access.log | sort | uniq -c | \
    awk '{printf "%s: %d requests (%.1f%%)\n", $2, $1, $1*100/total}' \
    total=$(wc -l < access.log)

What Works

• Always quote regex patterns with special characters. grep "$pattern" prevents the shell from expanding * or ? before grep sees them.
• Use awk for columnar data instead of cut when fields vary in width. cut fails on variable spacing; awk splits on any whitespace by default.
• Prefer jq for JSON, xq for XML, csvkit for CSV. Classic tools treat these formats as plain text and will break on quoted fields or nested structures.
• Chain tools left to right in order of filtering. Put grep early to reduce data volume before expensive awk or sort operations.
• Use LC_ALL=C for consistent sorting and performance. Locale-aware collation is slower and varies across systems; C gives you byte-wise order-dependent behavior.

Performance Tips

• Piping output into while read for per-line handling — the Bash Loop Over Files recipe covers the iteration side.

1. Use LC_ALL=C for 2-5x speedup on large files. Byte-wise sorting and comparison bypass locale-aware Unicode processing:

# Slow: locale-aware sorting
sort large_file.txt > sorted.txt

# Fast: byte-wise sorting (2-5x faster on ASCII data)
LC_ALL=C sort large_file.txt > sorted.txt

# Fast grep on large files
LC_ALL=C grep -n "pattern" huge_file.log

2. Use grep -F for fixed strings. When you do not need regex, fixed-string mode is considerably faster:

# Slow: regex engine processes literal string
grep "192.168.1.1" access.log

# Fast: fixed-string search (no regex overhead)
grep -F "192.168.1.1" access.log

# Fast: multiple fixed strings from a file
grep -Ff patterns.txt access.log

3. Use mawk or gawk for performance. mawk is faster than gawk for simple tasks, while gawk has more features:

# Check which awk is installed
awk --version 2>/dev/null || awk -W version 2>&1 | head -1

# Install mawk for speed (Debian/Ubuntu)
# apt-get install mawk

# Use mawk explicitly for performance-critical pipelines
mawk '{print $1, $9}' access.log | sort | uniq -c | sort -rn

For parallelizing these pipelines across CPU cores, see Bash Parallel Execution.

Common Mistakes

• Parsing JSON/HTML with grep/sed/awk. These are not structured formats — use jq, python -m json.tool, or a DOM parser.
• Forgetting that sed and awk operate line by line by default. Multi-line patterns require special flags (sed -z, awk RS manipulation) that are non-obvious.
• Assuming sort is stable by default. sort stability varies by implementation; use sort -s if you need it.
• Using cat unnecessarily. cat file | grep pattern is a useless use of cat. Use grep pattern file.
• Not handling empty input. Many pipelines fail silently on empty files — add | cat at the end or check file size first.

Further Reading

• GNU sed manual — address ranges and in-place editing
• GNU awk manual — fields, records, pattern-action rules
• BashFAQ/021 — when to use sed vs awk vs perl

Enable color output in grep

grep —color=auto “pattern” file.txt

Force color even when piping (useful for logging)

grep —color=always “ERROR” app.log | less -R

Custom color with awk

awk ’ /ERROR/ {print “\033[31m” $0 “\033[0m”; next} /WARN/ {print “\033[33m” $0 “\033[0m”; next} /INFO/ {print “\033[32m” $0 “\033[0m”; next} {print} ’ app.log