StackPractices
intermediate By Mathias Paulenko

Bash Backup Rotation: Retention and Integrity

Rotate Bash backups into daily, weekly, and monthly archives with retention, encryption, integrity checks, and optional S3 storage.

Overview

A scheduled job can leave you with no usable copy if it writes a broken archive or prunes the last good one. This recipe creates dated tar.gz files, keeps daily, weekly, and monthly tiers, reads each new archive before publishing it, and only then removes expired files.

When to Use

This rotation is a good fit for a Linux host that needs to:

  • create a compressed copy of a directory on a schedule;
  • keep daily recovery points and promote selected copies to weekly or monthly tiers;
  • stop before cleanup if archive creation or validation fails; or
  • add GPG encryption or remote S3 storage to the same workflow.

Tar isn’t a substitute for a database-native dump or filesystem snapshot when an application must be captured in a transactionally consistent state.

Solution

Bash backup rotation script

#!/usr/bin/env bash
set -euo pipefail

SOURCE_DIR="${1:-/var/app/data}"
BACKUP_DIR="${2:-/var/backups/app}"
DAILY_RETENTION=7
WEEKLY_RETENTION=4
MONTHLY_RETENTION=3

if [[ ! -d "$SOURCE_DIR" ]]; then
    echo "ERROR: Source directory doesn't exist: $SOURCE_DIR" >&2
    exit 1
fi

mkdir -p "$BACKUP_DIR/daily" "$BACKUP_DIR/weekly" "$BACKUP_DIR/monthly"

TODAY=$(date +%Y-%m-%d)
DAY_OF_WEEK=$(date +%u)
DAY_OF_MONTH=$(date +%d)

ARCHIVE="$BACKUP_DIR/daily/app-$TODAY.tar.gz"
TEMP_ARCHIVE=$(mktemp "$BACKUP_DIR/daily/.app-$TODAY.XXXXXX")
trap 'rm -f -- "$TEMP_ARCHIVE"' EXIT

# Build and read-test a temporary archive before replacing today's copy
tar -czf "$TEMP_ARCHIVE" -C "$(dirname "$SOURCE_DIR")" "$(basename "$SOURCE_DIR")"
tar -tzf "$TEMP_ARCHIVE" > /dev/null
mv -f -- "$TEMP_ARCHIVE" "$ARCHIVE"
trap - EXIT

# Promote to weekly on Sundays
if [[ "$DAY_OF_WEEK" == "7" ]]; then
    cp "$ARCHIVE" "$BACKUP_DIR/weekly/app-week-$TODAY.tar.gz"
fi

# Promote to monthly on the first day of the month
if [[ "$DAY_OF_MONTH" == "01" ]]; then
    cp "$ARCHIVE" "$BACKUP_DIR/monthly/app-month-$TODAY.tar.gz"
fi

# Remove old backups
find "$BACKUP_DIR/daily" -maxdepth 1 -type f -mtime "+$DAILY_RETENTION" -delete
find "$BACKUP_DIR/weekly" -maxdepth 1 -type f -mtime "+$((WEEKLY_RETENTION * 7))" -delete
find "$BACKUP_DIR/monthly" -maxdepth 1 -type f -mtime "+$((MONTHLY_RETENTION * 30))" -delete

echo "Backup completed: $ARCHIVE"

Explanation

This tiered daily/weekly/monthly scheme is the classic GFS (grandfather-father-son) rotation: the son tier keeps recent granular copies, and weekly/monthly archives keep long-range recovery points.

The script writes to a temporary file in the daily directory, then lists the archive with tar -tzf before moving it into place. Since the temporary and final files share a filesystem, the rename doesn’t expose a half-written daily copy. With set -euo pipefail, a failed create, test, or promotion exits before retention runs.

Sunday and first-of-month runs copy the validated daily archive into their longer-term tiers. These are full copies, not deduplicated snapshots, so budget the extra space. The find -mtime thresholds count full 24-hour periods: “7 days” is an age cutoff, not a promise to keep seven calendar dates.

flowchart diagram: Create temporary archive

Variants

Companion files: all these scripts live in the bash-backup-rotation-script companion repo, one runnable .sh per variant.

TierPromotionRetention settingStorage effect
DailyEvery successful runDAILY_RETENTION daysOne compressed archive per run date
WeeklyCopy on SundayWEEKLY_RETENTION × 7 daysFull extra copy
MonthlyCopy on day 1MONTHLY_RETENTION × 30 daysApproximate months; full extra copy
RemoteUpload after local validationS3 lifecycle policySeparate remote copy and storage charges

The settings use file age, not a count of calendar dates. If a strict “keep exactly N copies” policy is required, rotate by sorted archive names and test boundary dates explicitly.

Encrypted backup with GPG

#!/usr/bin/env bash
set -euo pipefail

SOURCE_DIR="${1:-/var/app/data}"
BACKUP_DIR="${2:-/var/backups/app}"
GPG_RECIPIENT="${3:-admin@example.com}"
RETENTION_DAYS=30

# Replace the placeholder with the fingerprint of the intended public key.
mkdir -p "$BACKUP_DIR/encrypted"
TODAY=$(date +%Y-%m-%d)
ARCHIVE="$BACKUP_DIR/encrypted/app-$TODAY.tar.gz.gpg"
TEMP_ARCHIVE=$(mktemp "$BACKUP_DIR/encrypted/.app-$TODAY.XXXXXX")
trap 'rm -f -- "$TEMP_ARCHIVE"' EXIT

# Create and encrypt backup in one pipeline (no unencrypted file on disk)
tar -czf - -C "$(dirname "$SOURCE_DIR")" "$(basename "$SOURCE_DIR")" | \
    gpg --encrypt --recipient "$GPG_RECIPIENT" --output "$TEMP_ARCHIVE"

# GPG can parse the packet headers here; this isn't a restore test.
if ! gpg --list-packets "$TEMP_ARCHIVE" > /dev/null 2>&1; then
    echo "ERROR: Encrypted backup verification failed" >&2
    exit 1
fi
mv -f -- "$TEMP_ARCHIVE" "$ARCHIVE"
trap - EXIT

echo "[$(date -Iseconds)] Encrypted backup created: $ARCHIVE"

# Prune old encrypted files only after the new archive passed the packet check.
find "$BACKUP_DIR/encrypted" -name '*.gpg' -type f -mtime "+$RETENTION_DAYS" -delete

# Decrypt for restore:
# gpg --decrypt app-2026-01-15.tar.gz.gpg | tar -xzf - -C /restore/path

Incremental backup with rsync

#!/usr/bin/env bash
set -euo pipefail

SOURCE_DIR="${1:-/var/app/data}"
BACKUP_DIR="${2:-/var/backups/rsync}"
SNAPSHOT_COUNT=7

mkdir -p "$BACKUP_DIR"

TODAY=$(date +%Y-%m-%d)
CURRENT_SNAPSHOT="$BACKUP_DIR/snapshot-$TODAY"

# Snapshot names use ISO dates, so lexical order is chronological. The first
# run has no --link-dest; each destination is a dedicated snapshot directory.
LAST_SNAPSHOT=""
for snapshot in "$BACKUP_DIR"/snapshot-*/; do
    [[ -d "$snapshot" ]] || continue
    snapshot="${snapshot%/}"
    if [[ "$snapshot" != "$CURRENT_SNAPSHOT" ]]; then
        if [[ -z "$LAST_SNAPSHOT" || "$snapshot" > "$LAST_SNAPSHOT" ]]; then
            LAST_SNAPSHOT="$snapshot"
        fi
    fi
done

RSYNC_ARGS=(-a --delete)
if [[ -n "$LAST_SNAPSHOT" ]]; then
    RSYNC_ARGS+=(--link-dest="$LAST_SNAPSHOT")
fi
rsync "${RSYNC_ARGS[@]}" "$SOURCE_DIR/" "$CURRENT_SNAPSHOT/"

# Verify snapshot integrity
SNAPSHOT_SIZE=$(du -sh "$CURRENT_SNAPSHOT" | cut -f1)
FILE_COUNT=$(find "$CURRENT_SNAPSHOT" -type f | wc -l)
echo "[$(date -Iseconds)] Snapshot created: $CURRENT_SNAPSHOT ($FILE_COUNT files, $SNAPSHOT_SIZE)"

# Rotate old dated snapshots without word-splitting paths that contain spaces.
snapshots=()
snapshot_count=0
for snapshot in "$BACKUP_DIR"/snapshot-*/; do
    [[ -d "$snapshot" ]] || continue
    snapshots[snapshot_count]="${snapshot%/}"
    snapshot_count=$((snapshot_count + 1))
done

if (( snapshot_count > SNAPSHOT_COUNT )); then
    remove_count=$((snapshot_count - SNAPSHOT_COUNT))
    for ((i = 0; i < remove_count; i++)); do
        snapshot="${snapshots[i]}"
        rm -r -- "$snapshot"
        echo "[$(date -Iseconds)] Removed old snapshot: $snapshot"
    done
fi

# Disk usage report
TOTAL_SIZE=$(du -sh "$BACKUP_DIR" | cut -f1)
echo "[$(date -Iseconds)] Total backup size: $TOTAL_SIZE"

S3 upload with checksum validation

The command sends S3 a SHA-256 checksum, and S3 rejects the upload if it doesn’t match. The created and host metadata are separate searchable fields, not checksum values. For profile setup, see AWS CLI Bash Scripts.

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="${1:-/var/backups/app}"
S3_BUCKET="${2:-my-app-backups}"
S3_PREFIX="${3:-daily}"

TODAY=$(date +%Y-%m-%d)
ARCHIVE="$BACKUP_DIR/daily/app-$TODAY.tar.gz"

# The AWS CLI sends a SHA-256 checksum; S3 rejects mismatches.
# User metadata is a separate field, not the checksum.
aws s3 cp "$ARCHIVE" "s3://$S3_BUCKET/$S3_PREFIX/app-$TODAY.tar.gz" \
    --checksum-algorithm SHA256 \
    --metadata "created=$(date -I),host=$(hostname)" \
    --no-progress

# List recent backups in S3
echo "=== Recent S3 Backups ==="
aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX/" --recursive | tail -10

Configure an S3 lifecycle policy separately from the daily upload job. The put-bucket-lifecycle-configuration API replaces the bucket’s entire lifecycle configuration; read and merge existing rules before applying a new one.

Backup integrity verification

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="${1:-/var/backups/app}"

echo "[$(date -Iseconds)] Starting backup verification..."

FAILURES=0
for archive in "$BACKUP_DIR"/daily/*.tar.gz; do
    [[ -f "$archive" ]] || continue
    filename=$(basename "$archive")

    # Listing reads the compressed stream and tar headers; it isn't a restore test.
    if ! tar -tzf "$archive" > /dev/null 2>&1; then
        echo "FAIL: $filename — tar archive is corrupt"
        FAILURES=$((FAILURES + 1))
        continue
    fi

    # Check file count in archive
    FILE_COUNT=$(tar -tzf "$archive" | wc -l)
    ARCHIVE_SIZE=$(du -h "$archive" | cut -f1)
    echo "OK: $filename — $FILE_COUNT files, $ARCHIVE_SIZE"
done

if (( FAILURES > 0 )); then
    echo "[$(date -Iseconds)] Verification FAILED: $FAILURES corrupt archives"
    exit 1
fi
echo "[$(date -Iseconds)] All archive streams passed the listing check"

Python backup with shutil and tarfile

import shutil
import tarfile
from datetime import datetime
from pathlib import Path

class BackupManager:
    """Manage backup rotation with daily, weekly, and monthly retention."""

    def __init__(self, source: str, backup_dir: str,
                 daily_retention: int = 7,
                 weekly_retention: int = 4,
                 monthly_retention: int = 3):
        self.source = Path(source)
        self.backup_dir = Path(backup_dir)
        self.daily_retention = daily_retention
        self.weekly_retention = weekly_retention
        self.monthly_retention = monthly_retention

        for subdir in ['daily', 'weekly', 'monthly']:
            (self.backup_dir / subdir).mkdir(parents=True, exist_ok=True)

    def create_backup(self) -> Path:
        today = datetime.now().strftime('%Y-%m-%d')
        archive_path = self.backup_dir / 'daily' / f'app-{today}.tar.gz'

        with tarfile.open(archive_path, 'w:gz') as tar:
            tar.add(self.source, arcname=self.source.name)

        self._promote_backups(today)
        self._cleanup_old_backups()
        return archive_path

    def _promote_backups(self, today: str) -> None:
        now = datetime.now()
        if now.weekday() == 6:  # Sunday
            shutil.copy2(
                self.backup_dir / 'daily' / f'app-{today}.tar.gz',
                self.backup_dir / 'weekly' / f'app-week-{today}.tar.gz'
            )
        if now.day == 1:
            shutil.copy2(
                self.backup_dir / 'daily' / f'app-{today}.tar.gz',
                self.backup_dir / 'monthly' / f'app-month-{today}.tar.gz'
            )

    def _cleanup_old_backups(self) -> None:
        cutoff = datetime.now()
        for subdir, retention in [
            ('daily', self.daily_retention),
            ('weekly', self.weekly_retention * 7),
            ('monthly', self.monthly_retention * 30),
        ]:
            dir_path = self.backup_dir / subdir
            for archive in dir_path.glob('*.tar.gz'):
                age = (cutoff - datetime.fromtimestamp(archive.stat().st_mtime)).days
                if age > retention:
                    archive.unlink()
                    print(f"Removed old backup: {archive.name} ({age} days old)")

    def verify_backup(self, archive_path: Path) -> bool:
        try:
            with tarfile.open(archive_path, 'r:gz') as tar:
                members = tar.getmembers()
                return len(members) > 0
        except (tarfile.TarError, EOFError) as e:
            print(f"Backup verification failed: {e}")
            return False

# Usage
manager = BackupManager('/var/app/data', '/var/backups/app')
archive = manager.create_backup()
if manager.verify_backup(archive):
    print(f"Backup created and verified: {archive}")
else:
    print("Backup verification failed!")

Database backup with pg_dump

#!/usr/bin/env bash
set -euo pipefail

DB_NAME="myapp"
DB_USER="postgres"          # expects ~/.pgpass or PGUSER auth — no password prompt in cron
BACKUP_DIR="/backups/db"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DUMP_FILE="${BACKUP_DIR}/db_${TIMESTAMP}.sql.gz"

mkdir -p "${BACKUP_DIR}"

# pipefail makes the pipeline fail if pg_dump fails, not just gzip
pg_dump -U "${DB_USER}" -w "${DB_NAME}" | gzip > "${DUMP_FILE}"
gzip -t "${DUMP_FILE}"

# Keep last 14 days — only runs when the dump above succeeded
find "${BACKUP_DIR}" -name "db_*.sql.gz" -type f -mtime +14 -delete

echo "Database backup created: db_${TIMESTAMP}.sql.gz"

For MySQL use mysqldump --single-transaction instead — it produces a consistent InnoDB snapshot without the FLUSH TABLES WITH READ LOCK anti-pattern (a lock held by mysql -e is released as soon as that client exits).

Restore helper

#!/usr/bin/env bash
set -euo pipefail
# restore-backup.sh — extracts an archive into an empty dir for inspection

BACKUP_FILE="${1:?Usage: $0 <backup_file>}"
RESTORE_DIR="/tmp/restore"

mkdir -p "${RESTORE_DIR}"

if [[ "$BACKUP_FILE" == *.gpg ]]; then
    gpg -d "${BACKUP_FILE}" | tar -xzf - -C "${RESTORE_DIR}"
else
    tar -xzf "${BACKUP_FILE}" -C "${RESTORE_DIR}"
fi

echo "Restored to ${RESTORE_DIR} — verify the tree before copying anything over live data"

Scheduling with cron

# Run daily at 2 AM, append output to a log
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

# Alert by mail on failure
0 2 * * * /opt/scripts/backup.sh || echo "Backup failed" | mail -s "Backup Alert" admin@example.com

Pre-flight disk space check

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/backups"
SOURCE_DIR="/var/www/myapp"

mkdir -p "${BACKUP_DIR}"   # df fails on a missing dir — create it first

# df -kP is POSIX: kilobytes on GNU and BSD, one line of output
available_kb=$(df -kP "${BACKUP_DIR}" | awk 'NR==2 {print $4}')
required_kb=$(du -sk "${SOURCE_DIR}" | awk '{print $1}')
available_gb=$((available_kb / 1024 / 1024))

if [ "$available_kb" -lt "$required_kb" ]; then
    echo "ERROR: only ${available_gb}GB free in ${BACKUP_DIR}, source needs $((required_kb / 1024 / 1024))GB"
    exit 1
fi

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
tar -czf "${BACKUP_DIR}/backup_${TIMESTAMP}.tar.gz" -C "${SOURCE_DIR}" .
echo "Backup created with ${available_gb}GB free space"

Speeding up large archives

# pigz compresses across all cores (GNU tar only — bsdtar has no -I)
tar -I pigz -cf "${BACKUP_FILE}" -C "${SOURCE_DIR}" .

# Skip caches, logs, and temp files
tar -czf "${BACKUP_FILE}"     --exclude="*.log"     --exclude="*.tmp"     --exclude="cache/*"     --exclude="node_modules/*"     -C "${SOURCE_DIR}" .

# Don't let backups starve production I/O
ionice -c 3 tar -czf "${BACKUP_FILE}" -C "${SOURCE_DIR}" .

Best Practices

  1. Test restores, not just archive listings. A readable tarball can still contain the wrong files. Run a scheduled restore drill and compare the restored tree with the expected files and ownership.
  2. Keep a copy outside the source host. A second directory on the same disk disappears with the original during a disk failure; use a separate host or storage account.
  3. Encrypt before remote upload. Use a managed recipient key for GPG; don’t leave the example recipient address unchanged.
  4. Protect retention tiers from deletion. Object Lock or a separate append-only destination can reduce the impact of compromised host credentials.
  5. Alert on non-zero exits. Keep the run log and last-success timestamp where the scheduler can expose them; silence alone doesn’t prove a backup completed.

Common Mistakes

  1. Pruning before a new archive is valid. Publish and list-test today’s archive before running any cleanup.
  2. Storing backups on the same disk as the source. A disk failure destroys both data and backups.
  3. Using only a date when the job can run more than once a day. Add a run ID or time to avoid overwriting the earlier archive.
  4. Ignoring exit codes. If tar fails but cleanup continues, find -delete can remove older recovery points without a replacement. Keep set -euo pipefail and alert on non-zero runs.
  5. Running as root without need. A dedicated service account limits the damage from a compromised scheduler; grant it read access to the source and write access to the backup directory.

See Also

Frequently Asked Questions

How do I restore a backup?

Extract it into a new, empty directory with tar -xzf app-YYYY-MM-DD.tar.gz -C /restore/path. Compare the directory tree and ownership there before copying anything over live data.

Can I use rsync instead of tar?

Use rsync when most files stay unchanged; its --link-dest variant below shares those files between dated snapshots. Choose tar when each recovery point must be a portable, self-contained archive.

What should I do when files change during a backup?

Pause writes or take a filesystem/database snapshot first. Tar copies files one by one, so a live data directory can mix values from different moments. For databases, use the database's dump tool or snapshot procedure, then document the recovery path in your runbook.