StackPractices
intermediate By Mathias Paulenko

Log Rotation and Compression in Bash

Rotate, compress and prune application logs with logrotate configs and custom bash scripts — including zstd, copytruncate, disk-space checks and S3 shipping.

Topics: devops

Overview

logrotate is the standard Linux utility that rotates, compresses, and prunes log files — and it exists because a single runaway log can fill the disk and take the app down with it. This recipe covers two approaches: logrotate configs for hosts where it’s installed, and custom bash scripts for containers or minimal systems where it isn’t. The scripts below are parameterized, handle the file-descriptor trap that silently loses log lines, and include the variants most tutorials skip — zstd, S3 shipping, and app-level rotation.

Companion files: every script and logrotate config from this page lives in the bash-log-rotation companion repo, one runnable file per use case.

When to Use

  • You run a web server or application that writes to log files continuously.
  • You need logs compressed and pruned to keep disk usage bounded.
  • You ship compressed logs to cold storage or need retention windows for compliance.
  • Your environment can’t run logrotate (container, embedded, minimal image).

Skip it if your app already logs to journald or to Docker’s json-file driver with max-size/max-file configured — the platform rotates for you and stacking scripts on top adds failure modes.

Solution

Using logrotate

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    dateext
    dateformat -%Y%m%d-%H%M%S
    sharedscripts
    create 0644 myapp myapp
    postrotate
        # Runs ONCE for the whole block thanks to sharedscripts
        if [ -f /var/run/myapp.pid ]; then
            kill -HUP "$(cat /var/run/myapp.pid)"
        else
            systemctl reload myapp
        fi
    endscript
}

Test the configuration:

# Dry run — shows what would happen, changes nothing
logrotate -d /etc/logrotate.d/myapp

# Force rotation now
logrotate -f /etc/logrotate.d/myapp

Size-based rotation with logrotate

# /etc/logrotate.d/myapp-size
/var/log/myapp/*.log {
    size 50M
    rotate 10
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    create 0644 myapp myapp
    postrotate
        systemctl reload myapp
    endscript
}

Compression with zstd instead of gzip

# /etc/logrotate.d/myapp-zstd
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    compresscmd /usr/bin/zstd
    compressext .zst
    delaycompress
    missingok
    notifempty
    create 640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp
    endscript
}

zstd compresses 50-80MB/s per core vs gzip’s 10-15MB/s with a similar ratio — worth it for large volumes. On a 500MB log file: gzip -6 takes ~12s, zstd -3 ~2s.

copytruncate for processes without signal support

# /etc/logrotate.d/myapp-truncate
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    copytruncate
    missingok
    notifempty
}

copytruncate copies the log, then truncates it in place — the app keeps writing to the same fd. Trade-off: a brief window where log lines can be lost during the copy.

Custom rotation script (no logrotate)

#!/bin/bash
set -euo pipefail

LOG_DIR="${1:-/var/log/myapp}"
RETENTION_DAYS="${2:-14}"
MAX_SIZE_MB="${3:-10}"

mkdir -p "$LOG_DIR/archive"

for log in "$LOG_DIR"/*.log; do
    [ -f "$log" ] || continue
    size_mb=$(du -m "$log" | cut -f1)

    if [ "$size_mb" -gt "$MAX_SIZE_MB" ]; then
        base=$(basename "$log" .log)
        timestamp=$(date +%Y%m%d-%H%M%S)
        rotated="$LOG_DIR/archive/${base}-${timestamp}.log"

        mv "$log" "$rotated"
        gzip "$rotated"
        # Recreate the log so the app can keep writing — and if it
        # supports it, signal it to reopen the fd (nginx, most daemons)
        touch "$log"
        # kill -HUP "$(cat /var/run/myapp.pid)" 2>/dev/null || true
    fi
done

find "$LOG_DIR/archive" -name '*.gz' -type f -mtime "+$RETENTION_DAYS" -delete

echo "[$(date -Iseconds)] Rotation done in $LOG_DIR"

If the app can’t reopen its log (no signal support), prefer the copytruncate pattern instead: cp "$log" "$rotated"; : > "$log"; gzip "$rotated".

Signal-based rotation (running process)

#!/bin/bash
set -euo pipefail

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
PID_FILE="/var/run/myapp.pid"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

mv "$LOG_FILE" "${LOG_DIR}/app_${TIMESTAMP}.log"
touch "$LOG_FILE"

if [ -f "$PID_FILE" ]; then
    kill -HUP "$(cat "$PID_FILE")"
fi

# Give the app a moment to start writing to the new file
sleep 5
gzip "${LOG_DIR}/app_${TIMESTAMP}.log"

find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +14 -delete

Parallel compression with pigz

#!/bin/bash
set -euo pipefail

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

[ -f "$LOG_FILE" ] || exit 0

ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"
mv "$LOG_FILE" "$ROTATED"
touch "$LOG_FILE"

# pigz uses all cores — ~2x faster than gzip on multi-core
if command -v pigz >/dev/null; then
    pigz "$ROTATED"
else
    gzip "$ROTATED"
fi

find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +14 -delete

Rotation with disk-space check

#!/bin/bash
set -euo pipefail

LOG_DIR="/var/log/myapp"
LOG_FILE="${LOG_DIR}/app.log"
RETENTION=14

# df column 5 is Use% (how FULL the disk is). -P keeps it on one line
USED_PERCENT=$(df -P "$LOG_DIR" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')

if [ "$USED_PERCENT" -gt 90 ]; then
    echo "CRITICAL: disk at ${USED_PERCENT}% — pruning harder"
    RETENTION=3
fi

[ -f "$LOG_FILE" ] || exit 0

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
ROTATED="${LOG_DIR}/app_${TIMESTAMP}.log"
mv "$LOG_FILE" "$ROTATED"
touch "$LOG_FILE"
gzip "$ROTATED"

find "$LOG_DIR" -name "app_*.log.gz" -type f -mtime +$RETENTION -delete

# Alert if the disk is still nearly full after rotation
USED_PERCENT=$(df -P "$LOG_DIR" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
if [ "$USED_PERCENT" -gt 90 ]; then
    echo "WARNING: disk still at ${USED_PERCENT}% after rotation"
fi

Email notification on rotation

# /etc/logrotate.d/myapp-notify
# Requires mailutils (Debian) or mailx (RHEL) — postrotate runs once
# thanks to sharedscripts, so you get one email, not one per log file
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    create 0644 myapp myapp
    postrotate
        systemctl reload myapp
        echo "Log rotated on $(hostname) at $(date)" | \
            mail -s "Log rotation: myapp" ops@example.com
    endscript
}

Cron entry for the custom script:

# Rotate daily at 4 AM, log the output
0 4 * * * /opt/scripts/log-rotate.sh >> /var/log/rotate.log 2>&1

Shipping rotated logs to S3

#!/bin/bash
set -euo pipefail

LOG_DIR="${1:-/var/log/myapp}"
S3_BUCKET="${2:-my-app-logs}"
S3_PREFIX="logs/$(hostname)"
RETENTION_DAYS="${4:-90}"

# Upload archives older than a day (already rotated and compressed)
find "$LOG_DIR/archive" -name '*.gz' -type f -mtime +1 | while read -r archive; do
    rel_path="${archive#$LOG_DIR/archive/}"
    if aws s3 cp "$archive" "s3://$S3_BUCKET/$S3_PREFIX/$rel_path" \
        --storage-class STANDARD_IA --no-progress; then
        echo "[$(date -Iseconds)] Uploaded $rel_path"
        rm "$archive"
    else
        echo "[$(date -Iseconds)] FAILED $rel_path"
    fi
done

# Local copies past retention get deleted; a bucket lifecycle
# rule moves them to cheaper storage on S3
find "$LOG_DIR/archive" -name '*.gz' -type f -mtime "+$RETENTION_DAYS" -delete

App-level rotation in Python

For Python apps, RotatingFileHandler rotates in-process — no external tool needed. This subclass adds gzip compression:

import logging, gzip, os
from logging.handlers import RotatingFileHandler

class CompressedRotatingFileHandler(RotatingFileHandler):
    def rotate(self, source, dest):
        super().rotate(source, dest)
        if os.path.exists(dest):
            with open(dest, 'rb') as f_in:
                with gzip.open(f'{dest}.gz', 'wb') as f_out:
                    f_out.writelines(f_in)
            os.remove(dest)

handler = CompressedRotatingFileHandler(
    '/var/log/app/application.log',
    maxBytes=100*1024*1024,  # 100MB
    backupCount=30,
)
logger = logging.getLogger('app')
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

Explanation

logrotate runs daily — via logrotate.timer on systemd distros, /etc/cron.daily/logrotate on older ones — and processes every config in /etc/logrotate.d/. Key directives:

  • daily/weekly/monthly/size N: when to rotate. size wins if reached first; minsize/maxsize add bounds.
  • rotate N: keep N archives before deleting.
  • compress/delaycompress: gzip archives; delaycompress skips compression for one cycle so a process still holding the old fd finishes.
  • sharedscripts: run postrotate once per block instead of once per matched file — without it, a *.log wildcard re-sends the HUP (or the email) for every file.
  • dateext/dateformat: timestamped archive names instead of .1, .2 — prevents overwriting a same-day second rotation.
  • su/create: required when rotating logs owned by a different user (the classic “parent directory has insecure permissions” error).
Decision flow for log rotation: if the host has logrotate use it — sharedscripts + postrotate HUP when the app supports signals, copytruncate when it doesn't; if it doesn't have logrotate, use a custom bash script or rotate in-app with RotatingFileHandler

The custom-script approach checks size, mvs the log, recreates it with touch, and compresses — the order matters: recreating (or signaling) is what stops the app from writing into the renamed inode and losing lines. The scripts use GNU coreutils flags (du -m, stat -c%s); on BSD/macOS swap stat -c%s for stat -f%z.

Variants

ApproachToolCompressionUse When
logrotatelogrotategzip / zstdStandard Linux servers
copytruncatelogrotategzipApps that can’t reopen the log fd
Custom size-basedbash + statgzip / pigzContainers, minimal systems
Signal-basedbash + killgzipDaemons with HUP support
S3 shippingbash + aws cligzipCompliance retention, cold storage
In-app handlerPython logginggzipApps that should self-rotate

Best Practices

  • Prefer logrotate when available — it handles permissions, su, and signals for you.
  • Keep a dry-run habit: logrotate -d before any config change, and test custom scripts against a copy of the logs first — a bad find -delete eats live data.
  • Rotate at 70-80% disk usage, not 99% — monitor it (see Monitoring Disk Usage with Bash).
  • Restrict archive permissions (chmod 640) — logs can contain sensitive data.
  • Ship archives to cold storage for anything past compliance windows; pair with the backup rotation script for a full lifecycle.

Common Mistakes

  • mv + gzip without recreating the log or signaling the app — the process keeps writing to the moved inode and the lines are gone. This is the single most common log-loss bug.
  • Wildcard configs without sharedscripts — postrotate fires once per matched file, so the app gets reloaded (or an email sent) for every log.
  • copytruncate on huge logs — it copies the whole file first and briefly doubles disk usage; and a burst of writes mid-copy is lost.
  • Timestamped names like date +%Y%m%d without a time component — a second rotation the same day silently overwrites the first.
  • df without -P — long device names wrap the output and awk reads the wrong column.

Performance Tips

  • zstd over gzip for large volumes: zstd -3 ~50-80MB/s per core vs gzip -6 ~10-15MB/s, same ballpark ratio — it’s what Facebook, Amazon and Cloudflare ship.
  • pigz when gzip is required: same format, all cores — ~2x faster on a typical 4-core box.
  • Lower the level before switching tools: gzip -1 is roughly 5x faster than -9 for a modest ratio loss; with logrotate use compressoptions -1.
  • Rotate off-peak: cron at 4 AM, not midnight — every admin in the world schedules midnight.

See Also

Frequently Asked Questions

What is the difference between postrotate and prerotate?

prerotate runs before the log is rotated; postrotate runs after. Use prerotate to stop a service cleanly and postrotate to reload it so it reopens the new log file. Both close with endscript, and under a wildcard pattern you want sharedscripts so the block runs once, not per file.

How do I rotate logs in a Docker container?

Container stdout/stderr goes to Docker's json-file driver — configure rotation in /etc/docker/daemon.json:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "5"
    }
}

For apps that write log files inside the container, mount the log dir as a volume and run logrotate on the host:

docker run -v /var/log/myapp:/var/log/myapp myapp
How do I handle multiple log files with different policies?

Create one config per file (or pattern) in /etc/logrotate.d/ — each block has its own frequency, retention, and postrotate. Don't lump them into one wildcard block unless every file should share the same policy.

# /etc/logrotate.d/myapp-access
/var/log/myapp/access.log { daily rotate 7 compress }

# /etc/logrotate.d/myapp-error
/var/log/myapp/error.log { weekly rotate 52 compress }
Is this solution production-ready?

Yes — logrotate ships with Ubuntu, Debian, RHEL, CentOS and Amazon Linux. The custom scripts use only coreutils (mv, gzip, find, touch). zstd runs at Facebook, Amazon and Cloudflare, and the S3-shipping pattern is what teams use for 1-7 year compliance retention.

How do I debug a rotation that isn't working?

Run logrotate -d for a dry run and logrotate -v to watch it happen — the classic failure is "parent directory has insecure permissions" (fix with su). For custom scripts add set -x; verify archives with gzip -t/zstd -t; check the app actually reopened the fd with ls -l /proc/<pid>/fd. If nothing rotated at all, check the state file /var/lib/logrotate/status.

What does copytruncate do?

copytruncate copies the current log to a rotated name, then truncates the original in place — the app keeps writing to the same file descriptor, so no signal is needed. The trade-off: a brief window during the copy where log lines can be lost, and doubled disk usage while copying.