Log Rotation and Compression in Bash
Rotate, compress and prune application logs with logrotate configs and custom bash scripts — including zstd, copytruncate, disk-space checks and S3 shipping.
Overview
logrotate is the standard Linux utility that rotates, compresses, and prunes log files — and it exists because a single runaway log can fill the disk and take the app down with it. This recipe covers two approaches: logrotate configs for hosts where it’s installed, and custom bash scripts for containers or minimal systems where it isn’t. The scripts below are parameterized, handle the file-descriptor trap that silently loses log lines, and include the variants most tutorials skip — zstd, S3 shipping, and app-level rotation.
Companion files: every script and logrotate config from this page lives in the bash-log-rotation companion repo, one runnable file per use case.
When to Use
- You run a web server or application that writes to log files continuously.
- You need logs compressed and pruned to keep disk usage bounded.
- You ship compressed logs to cold storage or need retention windows for compliance.
- Your environment can’t run logrotate (container, embedded, minimal image).
Skip it if your app already logs to journald or to Docker’s json-file driver with max-size/max-file configured — the platform rotates for you and stacking scripts on top adds failure modes.
Solution
Using logrotate
# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
dateext
dateformat -%Y%m%d-%H%M%S
sharedscripts
create 0644 myapp myapp
postrotate
# Runs ONCE for the whole block thanks to sharedscripts
if [ -f /var/run/myapp.pid ]; then
kill -HUP "$(cat /var/run/myapp.pid)"
else
systemctl reload myapp
fi
endscript
}
Test the configuration:
# Dry run — shows what would happen, changes nothing
logrotate -d /etc/logrotate.d/myapp
# Force rotation now
logrotate -f /etc/logrotate.d/myapp
Size-based rotation with logrotate
# /etc/logrotate.d/myapp-size
/var/log/myapp/*.log {
size 50M
rotate 10
compress
delaycompress
missingok
notifempty
sharedscripts
create 0644 myapp myapp
postrotate
systemctl reload myapp
endscript
}
Compression with zstd instead of gzip
# /etc/logrotate.d/myapp-zstd
/var/log/myapp/*.log {
daily
rotate 30
compress
compresscmd /usr/bin/zstd
compressext .zst
delaycompress
missingok
notifempty
create 640 myapp myapp
sharedscripts
postrotate
systemctl reload myapp
endscript
}
zstd compresses 50-80MB/s per core vs gzip’s 10-15MB/s with a similar ratio — worth it for large volumes. On a 500MB log file: gzip -6 takes ~12s, zstd -3 ~2s.
copytruncate for processes without signal support
# /etc/logrotate.d/myapp-truncate
/var/log/myapp/*.log {
daily
rotate 14
compress
copytruncate
missingok
notifempty
}
copytruncate copies the log, then truncates it in place — the app keeps writing to the same fd. Trade-off: a brief window where log lines can be lost during the copy.
Custom rotation script (no logrotate)
#!/bin/bash
set -euo pipefail
LOG_DIR="${1:-/var/log/myapp}"
RETENTION_DAYS="${2:-14}"
MAX_SIZE_MB="${3:-10}"
mkdir -p "$LOG_DIR/archive"
for log in "$LOG_DIR"/*.log; do
[ -f "$log" ] || continue
size_mb=$(du -m "$log" | cut -f1)
if [ "$size_mb" -gt "$MAX_SIZE_MB" ]; then
base=$(basename "$log" .log)
timestamp=$(date +%Y%m%d-%H%M%S)
rotated="$LOG_DIR/archive/${base}-${timestamp}.log"
mv "$log" "$rotated"
gzip "$rotated"
# Recreate the log so the app can keep writing — and if it
# supports it, signal it to reopen the fd (nginx, most daemons)
touch "$log"
# kill -HUP "$(cat /var/run/myapp.pid)" 2>/dev/null || true
fi
done
find "$LOG_DIR/archive" -name '*.gz' -type f -mtime "+$RETENTION_DAYS" -delete
echo "[$(date -Iseconds)] Rotation done in $LOG_DIR"
If the app can’t reopen its log (no signal support), prefer the copytruncate pattern instead: cp "$log" "$rotated"; : > "$log"; gzip "$rotated".
Signal-based rotation (running process)
#!/bin/bash
set -euo pipefail
LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
PID_FILE="/var/run/myapp.pid"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
mv "$LOG_FILE" "${LOG_DIR}/app_${TIMESTAMP}.log"
touch "$LOG_FILE"
if [ -f "$PID_FILE" ]; then
kill -HUP "$(cat "$PID_FILE")"
fi
# Give the app a moment to start writing to the new file
sleep 5
gzip "${LOG_DIR}/app_${TIMESTAMP}.log"
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +14 -delete
Parallel compression with pigz
#!/bin/bash
set -euo pipefail
LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
[ -f "$LOG_FILE" ] || exit 0
ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"
mv "$LOG_FILE" "$ROTATED"
touch "$LOG_FILE"
# pigz uses all cores — ~2x faster than gzip on multi-core
if command -v pigz >/dev/null; then
pigz "$ROTATED"
else
gzip "$ROTATED"
fi
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +14 -delete
Rotation with disk-space check
#!/bin/bash
set -euo pipefail
LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
RETENTION=14
# df column 5 is Use% (how FULL the disk is). -P keeps it on one line
USED_PERCENT=$(df -P "$LOG_DIR" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
if [ "$USED_PERCENT" -gt 90 ]; then
echo "CRITICAL: disk at ${USED_PERCENT}% — pruning harder"
RETENTION=3
fi
[ -f "$LOG_FILE" ] || exit 0
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"
mv "$LOG_FILE" "$ROTATED"
touch "$LOG_FILE"
gzip "$ROTATED"
find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +$RETENTION -delete
# Alert if the disk is still nearly full after rotation
USED_PERCENT=$(df -P "$LOG_DIR" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
if [ "$USED_PERCENT" -gt 90 ]; then
echo "WARNING: disk still at ${USED_PERCENT}% after rotation"
fi
Email notification on rotation
# /etc/logrotate.d/myapp-notify
# Requires mailutils (Debian) or mailx (RHEL) — postrotate runs once
# thanks to sharedscripts, so you get one email, not one per log file
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
sharedscripts
create 0644 myapp myapp
postrotate
systemctl reload myapp
echo "Log rotated on $(hostname) at $(date)" | \
mail -s "Log rotation: myapp" ops@example.com
endscript
}
Cron entry for the custom script:
# Rotate daily at 4 AM, log the output
0 4 * * * /opt/scripts/log-rotate.sh >> /var/log/rotate.log 2>&1
Shipping rotated logs to S3
#!/bin/bash
set -euo pipefail
LOG_DIR="${1:-/var/log/myapp}"
S3_BUCKET="${2:-my-app-logs}"
S3_PREFIX="logs/$(hostname)"
RETENTION_DAYS="${4:-90}"
# Upload archives older than a day (already rotated and compressed)
find "$LOG_DIR/archive" -name '*.gz' -type f -mtime +1 | while read -r archive; do
rel_path="${archive#$LOG_DIR/archive/}"
if aws s3 cp "$archive" "s3://$S3_BUCKET/$S3_PREFIX/$rel_path" \
--storage-class STANDARD_IA --no-progress; then
echo "[$(date -Iseconds)] Uploaded $rel_path"
rm "$archive"
else
echo "[$(date -Iseconds)] FAILED $rel_path"
fi
done
# Local copies past retention get deleted; a bucket lifecycle
# rule moves them to cheaper storage on S3
find "$LOG_DIR/archive" -name '*.gz' -type f -mtime "+$RETENTION_DAYS" -delete
App-level rotation in Python
For Python apps, RotatingFileHandler rotates in-process — no external tool needed. This subclass adds gzip compression:
import logging, gzip, os
from logging.handlers import RotatingFileHandler
class CompressedRotatingFileHandler(RotatingFileHandler):
def rotate(self, source, dest):
super().rotate(source, dest)
if os.path.exists(dest):
with open(dest, 'rb') as f_in:
with gzip.open(f'{dest}.gz', 'wb') as f_out:
f_out.writelines(f_in)
os.remove(dest)
handler = CompressedRotatingFileHandler(
'/var/log/app/application.log',
maxBytes=100*1024*1024, # 100MB
backupCount=30,
)
logger = logging.getLogger('app')
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
Explanation
logrotate runs daily — via logrotate.timer on systemd distros, /etc/cron.daily/logrotate on older ones — and processes every config in /etc/logrotate.d/. Key directives:
daily/weekly/monthly/size N: when to rotate.sizewins if reached first;minsize/maxsizeadd bounds.rotate N: keep N archives before deleting.compress/delaycompress: gzip archives;delaycompressskips compression for one cycle so a process still holding the old fd finishes.sharedscripts: runpostrotateonce per block instead of once per matched file — without it, a*.logwildcard re-sends the HUP (or the email) for every file.dateext/dateformat: timestamped archive names instead of.1,.2— prevents overwriting a same-day second rotation.su/create: required when rotating logs owned by a different user (the classic “parent directory has insecure permissions” error).
The custom-script approach checks size, mvs the log, recreates it with touch, and compresses — the order matters: recreating (or signaling) is what stops the app from writing into the renamed inode and losing lines. The scripts use GNU coreutils flags (du -m, stat -c%s); on BSD/macOS swap stat -c%s for stat -f%z.
Variants
| Approach | Tool | Compression | Use When |
|---|---|---|---|
| logrotate | logrotate | gzip / zstd | Standard Linux servers |
| copytruncate | logrotate | gzip | Apps that can’t reopen the log fd |
| Custom size-based | bash + stat | gzip / pigz | Containers, minimal systems |
| Signal-based | bash + kill | gzip | Daemons with HUP support |
| S3 shipping | bash + aws cli | gzip | Compliance retention, cold storage |
| In-app handler | Python logging | gzip | Apps that should self-rotate |
Best Practices
- Prefer
logrotatewhen available — it handles permissions,su, and signals for you. - Keep a dry-run habit:
logrotate -dbefore any config change, and test custom scripts against a copy of the logs first — a badfind -deleteeats live data. - Rotate at 70-80% disk usage, not 99% — monitor it (see Monitoring Disk Usage with Bash).
- Restrict archive permissions (
chmod 640) — logs can contain sensitive data. - Ship archives to cold storage for anything past compliance windows; pair with the backup rotation script for a full lifecycle.
Common Mistakes
mv+gzipwithout recreating the log or signaling the app — the process keeps writing to the moved inode and the lines are gone. This is the single most common log-loss bug.- Wildcard configs without
sharedscripts—postrotatefires once per matched file, so the app gets reloaded (or an email sent) for every log. copytruncateon huge logs — it copies the whole file first and briefly doubles disk usage; and a burst of writes mid-copy is lost.- Timestamped names like
date +%Y%m%dwithout a time component — a second rotation the same day silently overwrites the first. dfwithout-P— long device names wrap the output andawkreads the wrong column.
Performance Tips
zstdovergzipfor large volumes:zstd -3~50-80MB/s per core vsgzip -6~10-15MB/s, same ballpark ratio — it’s what Facebook, Amazon and Cloudflare ship.pigzwhen gzip is required: same format, all cores — ~2x faster on a typical 4-core box.- Lower the level before switching tools:
gzip -1is roughly 5x faster than-9for a modest ratio loss; with logrotate usecompressoptions -1. - Rotate off-peak: cron at 4 AM, not midnight — every admin in the world schedules midnight.
See Also
- Advanced iptables Firewall Rules — for the servers you’re hardening alongside logging
- Monitoring Disk Usage with Bash — alert before the disk that holds your logs fills up
- logrotate(8) man page — every directive, documented upstream
- Debian wiki: logrotate — distro-specific notes and examples
Frequently Asked Questions
What is the difference between postrotate and prerotate?
prerotate runs before the log is rotated; postrotate runs after. Use prerotate to stop a service cleanly and postrotate to reload it so it reopens the new log file. Both close with endscript, and under a wildcard pattern you want sharedscripts so the block runs once, not per file.
How do I rotate logs in a Docker container?
Container stdout/stderr goes to Docker's json-file driver — configure rotation in /etc/docker/daemon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "100m",
"max-file": "5"
}
}
For apps that write log files inside the container, mount the log dir as a volume and run logrotate on the host:
docker run -v /var/log/myapp:/var/log/myapp myapp
How do I handle multiple log files with different policies?
Create one config per file (or pattern) in /etc/logrotate.d/ — each block has its own frequency, retention, and postrotate. Don't lump them into one wildcard block unless every file should share the same policy.
# /etc/logrotate.d/myapp-access
/var/log/myapp/access.log { daily rotate 7 compress }
# /etc/logrotate.d/myapp-error
/var/log/myapp/error.log { weekly rotate 52 compress }
Is this solution production-ready?
Yes — logrotate ships with Ubuntu, Debian, RHEL, CentOS and Amazon Linux. The custom scripts use only coreutils (mv, gzip, find, touch). zstd runs at Facebook, Amazon and Cloudflare, and the S3-shipping pattern is what teams use for 1-7 year compliance retention.
How do I debug a rotation that isn't working?
Run logrotate -d for a dry run and logrotate -v to watch it happen — the classic failure is "parent directory has insecure permissions" (fix with su). For custom scripts add set -x; verify archives with gzip -t/zstd -t; check the app actually reopened the fd with ls -l /proc/<pid>/fd. If nothing rotated at all, check the state file /var/lib/logrotate/status.
What does copytruncate do?
copytruncate copies the current log to a rotated name, then truncates the original in place — the app keeps writing to the same file descriptor, so no signal is needed. The trade-off: a brief window during the copy where log lines can be lost, and doubled disk usage while copying.
Related Resources
Bash Backup Rotation: Retention and Integrity
Rotate Bash backups into daily, weekly, and monthly archives with retention, encryption, integrity checks, and optional S3 storage.
RecipeBash Scripting for DevOps Automation and System Tasks
How to write reliable Bash scripts for automating deployments, system monitoring, log rotation, and routine maintenance tasks
RecipeCompress and Decompress Files with ZIP, GZIP, and TAR
Create and extract ZIP, GZIP, and TAR archives in Python, Node.js, Java, and Bash, with streaming reads and safe-extraction patterns.
RecipeAnsible Playbook for Server Configuration
How to write and run Ansible playbooks for provisioning, configuring, and managing servers with idempotent tasks, roles, and inventory files.
RecipeAWS CLI Bash Scripts
A working set of Bash scripts that wrap the AWS CLI to provision, tag, audit, and clean up AWS resources, including multi-account deploys.
RecipeMonitor Disk Usage
Alert when disk space crosses thresholds using a Bash script that checks mount points and notifies operators.