Log Rotation and Compression in Bash

Overview

Log files grow without limit. Without rotation, a single log file can fill the disk and crash the application. Below is a practical approach to two approaches: using logrotate (the Linux standard) and writing a custom bash script for environments where logrotate is not available.

When to Use

• You run a web server or application that writes to log files continuously
• You need to keep logs compressed to save disk space
• You want automatic cleanup of old log files
• You are setting up log management on a new server

Solution

Using logrotate

# /etc/logrotate.d/myapp

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0644 myapp myapp
    postrotate
        systemctl reload myapp
    endscript
}

Test the configuration:

# Dry run
logrotate -d /etc/logrotate.d/myapp

# Force rotation
logrotate -f /etc/logrotate.d/myapp

Custom log rotation script

#!/bin/bash

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
MAX_SIZE=$((10 * 1024 * 1024))  # 10 MB
KEEP=14

# Check if rotation needed
if [ ! -f "$LOG_FILE" ]; then
    exit 0
fi

CURRENT_SIZE=$(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)

if [ "$CURRENT_SIZE" -lt "$MAX_SIZE" ]; then
    exit 0
fi

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"

mv "$LOG_FILE" "$ROTATED"
gzip "$ROTATED"

# Cleanup old rotated logs
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +${KEEP} -delete

echo "Rotated: ${ROTATED}.gz"

Time-based rotation with cron

#!/bin/bash

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
TIMESTAMP=$(date +%Y%m%d)

mv "$LOG_FILE" "${LOG_DIR}/app_${TIMESTAMP}.log"
gzip "${LOG_DIR}/app_${TIMESTAMP}.log"
touch "$LOG_FILE"

# Keep 30 days
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +30 -delete

Cron entry:

# Rotate daily at midnight
0 0 * * * /opt/scripts/log-rotate.sh >> /var/log/rotate.log 2>&1

Signal-based rotation (for running processes)

#!/bin/bash

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
PID_FILE="/var/run/myapp.pid"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Move current log
mv "$LOG_FILE" "${LOG_DIR}/app_${TIMESTAMP}.log"

# Send HUP signal to reopen log file
if [ -f "$PID_FILE" ]; then
    kill -HUP $(cat "$PID_FILE")
fi

# Compress old log after a delay
sleep 60
gzip "${LOG_DIR}/app_${TIMESTAMP}.log"

# Cleanup
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +14 -delete

Log rotation with error handling

#!/bin/bash

set -euo pipefail

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
KEEP=14

mkdir -p "$LOG_DIR"

if [ ! -f "$LOG_FILE" ]; then
    echo "No log file to rotate"
    exit 0
fi

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"

mv "$LOG_FILE" "$ROTATED" || { echo "Failed to move log"; exit 1; }
gzip "$ROTATED" || { echo "Failed to compress log"; exit 1; }

find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +${KEEP} -delete

echo "Rotation complete: ${ROTATED}.gz"

Explanation

logrotate is the standard Linux tool for log management. It runs daily via cron (/etc/cron.daily/logrotate) and processes all configs in /etc/logrotate.d/. Key directives:

• daily/weekly/monthly: Rotation frequency.
• rotate N: Keep N rotated files before deleting.
• compress/delaycompress: Compress rotated logs. delaycompress waits one rotation cycle before compressing (useful for processes that still write to the old file).
• missingok: Don’t error if the log file doesn’t exist.
• postrotate/endscript: Run commands after rotation (e.g., reload the app to reopen log files).

The custom script approach uses stat to check file size and mv + gzip to rotate. It works in containers or minimal systems without logrotate installed.

Variants

Approach	Tool	Compression	Use When
logrotate	logrotate	gzip	Standard Linux servers
Custom size-based	bash + stat	gzip	Containers, minimal systems
Time-based cron	bash + cron	gzip	Simple setups, predictable schedules
Signal-based	bash + kill	gzip	Long-running processes with HUP support
logrotate with copytruncate	logrotate	gzip	Processes that don’t support signals

Guidelines

• Use logrotate when available. It handles edge cases, permissions, and signal sending.
• Use delaycompress with postrotate if your process keeps writing to the old file handle.
• Set missingok to avoid errors when the log file doesn’t exist yet.
• Test with logrotate -d (dry run) before applying to production.
• Monitor disk space after rotation. A spike means rotation isn’t running.

Common Mistakes

• Not sending a signal after rotation. The process keeps writing to the deleted file handle and logs disappear.
• Forgetting delaycompress. The process may still have the old file open when gzip tries to compress it.
• Not setting missingok. The script errors on first run when no log file exists.
• Using copytruncate without understanding it copies the entire file first, doubling disk usage briefly.
• Running rotation too frequently for small logs. Daily is enough for most applications.

Performance Tips

1. Use pigz instead of gzip for large logs. Parallel compression on multi-core systems:

# Install pigz
apt install pigz  # Debian/Ubuntu
yum install pigz  # RHEL/CentOS

# Use in custom scripts
pigz -4 "$ROTATED"  # Level 4, good balance of speed/ratio

2. Use lower compression levels for speed. Level 1 is 5x faster than level 9:

# Fast rotation with minimal compression
gzip -1 "$ROTATED"

# Or with logrotate
compressoptions -1

3. Run rotation during low-traffic periods. Schedule cron for off-peak hours:

# Rotate at 4 AM instead of midnight
0 4 * * * /usr/sbin/logrotate /etc/logrotate.d/myapp