StackPractices
beginner By Mathias Paulenko

Monitor Disk Usage

Alert when disk space crosses thresholds using a Bash script that checks mount points and notifies operators.

Topics: devops

Overview

Disk space is a silent killer of production services. When a log, cache, or database fills the disk, applications crash, writes fail, and recovery becomes urgent. A simple Bash script that checks mount points and alerts on threshold breaches buys you the hours you need to react — and wired to a cleanup job, it frees space before the outage.

When to Use

Use this resource when:

  • You want disk alerts without deploying a full monitoring agent on the box.
  • Alerts have to reach email, Slack, or a log file when usage climbs.
  • Your fleet is containers, VMs, or bare-metal boxes where disk is tight.
  • You want the script to free space on its own once usage crosses a threshold.

Skip it if a monitoring agent is already deployed — Prometheus node_exporter, Datadog, or a cloud provider agent already ships disk alerts, and a second cron script only adds noise.

Solution

Disk usage monitoring script

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD="${1:-80}"
EMAIL="${2:-admin@example.com}"

# Check all local mount points
df -Hl | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs usage mount; do
    usage_val="${usage%\%}"
    if (( usage_val >= THRESHOLD )); then
        echo "WARNING: $mount ($fs) is at $usage"
        # Send alert (example with mail)
        echo "Disk usage on $mount is $usage" | mail -s "Disk alert: $mount" "$EMAIL"
    else
        echo "OK: $mount is at $usage"
    fi
done

# Optional: trigger cleanup for specific mounts
# if (( usage_val >= 90 )); then
#     /usr/local/bin/cleanup-logs.sh
# fi

Runnable files: every script on this page is in the bash-monitoring-disk-usage companion repo, including the systemd units and the Prometheus alert rules.

Explanation

The script runs df -Hl to get every local filesystem with its usage percentage. awk filters out the header and non-device entries. For each mount point, it strips the percent sign, compares the numeric value to the threshold, and prints a warning or OK message. Cross the threshold and the script mails an alert. The cleanup block stays commented out — audit what it deletes before you turn it on.

flowchart diagram: Cron or systemd timer

Variants

Alert channelToolBest for
EmailmailSimple servers with local MTA
Slackwebhook curlTeams already using Slack
PagerDutyevent APIProduction on-call escalation
File logredirect to syslogCentralized log aggregation

Slack alert integration

#!/usr/bin/env bash
set -euo pipefail

SLACK_WEBHOOK="${SLACK_WEBHOOK:-https://hooks.slack.com/services/XXX}"
THRESHOLD="${1:-80}"
HOSTNAME=$(hostname)

send_slack_alert() {
    local mount="$1" usage="$2" fs="$3"
    local payload
    payload=$(cat <<EOF
{
    "text": "Disk alert on ${HOSTNAME}",
    "attachments": [{
        "color": "danger",
        "fields": [
            {"title": "Mount", "value": "${mount}", "short": true},
            {"title": "Usage", "value": "${usage}", "short": true},
            {"title": "Filesystem", "value": "${fs}", "short": true},
            {"title": "Host", "value": "${HOSTNAME}", "short": true}
        ]
    }]
}
EOF
)
    curl -s -X POST -H 'Content-Type: application/json' -d "$payload" "$SLACK_WEBHOOK"
}

df -Hl | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs usage mount; do
    usage_val="${usage%\%}"
    if (( usage_val >= THRESHOLD )); then
        send_slack_alert "$mount" "$usage" "$fs"
        echo "[$(date -Iseconds)] WARNING: $mount ($fs) at $usage — Slack alert sent"
    fi
done

Per-partition thresholds

A single threshold rarely fits every mount — /var tolerates growth better than /. Define one threshold per partition:

#!/usr/bin/env bash
set -euo pipefail

ALERT_EMAIL="admin@example.com"

# Format: "mountpoint:threshold"
PARTITIONS=(
    "/:80"
    "/var:90"
    "/home:75"
    "/tmp:85"
)

for entry in "${PARTITIONS[@]}"; do
    mount="${entry%%:*}"
    threshold="${entry##*:}"

    usage_val=$(df -H "$mount" | awk 'NR>1 {print $5}' | tr -d '%')

    if (( usage_val >= threshold )); then
        echo "ALERT: $mount at ${usage_val}% (threshold: ${threshold}%)"
        echo "Partition $mount at ${usage_val}%" | mail -s "Disk alert: $mount" "$ALERT_EMAIL"
    fi
done

Alert only on state change

If the script runs every 5 minutes and a mount stays at 92%, you get an alert every 5 minutes. Keep a state file so each mount alerts once when it crosses the threshold, and clears when it recovers:

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD=80
STATE_FILE="${STATE_FILE:-/var/run/disk-alert-state}"
touch "$STATE_FILE"

df -Hl | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs usage mount; do
    usage_val="${usage%\%}"
    if (( usage_val >= THRESHOLD )) && ! grep -qx "$mount" "$STATE_FILE"; then
        echo "$mount" >> "$STATE_FILE"
        echo "Disk alert on $(hostname): $mount at $usage" | mail -s "Disk alert: $mount" admin@example.com
    elif (( usage_val < THRESHOLD )) && grep -qx "$mount" "$STATE_FILE"; then
        sed -i "\|^${mount}$|d" "$STATE_FILE"
        echo "RECOVERED: $mount back to $usage"
    fi
done

Inode monitoring (disk can have space but no inodes)

#!/usr/bin/env bash
set -euo pipefail

INODE_THRESHOLD="${1:-80}"

df -i | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs iusage mount; do
    iusage_val="${iusage%\%}"
    if (( iusage_val >= INODE_THRESHOLD )); then
        echo "WARNING: $mount ($fs) inode usage at $iusage"
        # Find directories with the most files
        top_dirs=$(find "$mount" -xdev -type d -exec sh -c 'echo $(find "$0" -maxdepth 1 -type f | wc -l) "$0"' {} \; 2>/dev/null | sort -rn | head -5)
        echo "Top directories by file count:"
        echo "$top_dirs"
    fi
done

Find what is consuming space

When an alert fires, the next question is which directories grew. This companion snippet lists the largest directories and files:

#!/usr/bin/env bash

echo "=== Top 10 largest directories ==="
du -sh /* 2>/dev/null | sort -rh | head -10

echo "=== Top 10 largest files ==="
find / -type f -size +100M -exec du -h {} + 2>/dev/null | sort -rh | head -10

Systemd timer for disk monitoring (replaces cron)

# /etc/systemd/system/disk-monitor.service
[Unit]
Description=Disk usage monitoring
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disk-monitor.sh 80 admin@example.com
StandardOutput=journal
StandardError=journal
# /etc/systemd/system/disk-monitor.timer
[Unit]
Description=Run disk monitor every 5 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
AccuracySec=30s

[Install]
WantedBy=timers.target
# Enable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now disk-monitor.timer
systemctl list-timers disk-monitor.timer

See the systemd.timer reference for the full calendar syntax. On hosts without systemd, a cron entry does the same job: */5 * * * * /usr/local/bin/disk-monitor.sh 80 admin@example.com.

Prometheus node_exporter disk alerts

If you already run Prometheus, skip the cron script and alert on node_filesystem_* metrics instead — including predict_linear, which fires before the disk is full. For the full server setup, see the Prometheus and Grafana guide.

# prometheus-alerts.yml
groups:
  - name: disk_usage
    rules:
      - alert: HighDiskUsage
        expr: |
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk usage high on {{ $labels.instance }}"
          description: "{{ $labels.mountpoint }} is at {{ printf \"%.1f\" $value }}% usage"

      - alert: CriticalDiskUsage
        expr: |
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Disk usage critical on {{ $labels.instance }}"
          description: "{{ $labels.mountpoint }} is at {{ printf \"%.1f\" $value }}% usage"

      - alert: HighInodeUsage
        expr: |
          (1 - node_filesystem_files_free / node_filesystem_files) * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Inode usage high on {{ $labels.instance }}"
          description: "{{ $labels.mountpoint }} has {{ printf \"%.1f\" $value }}% inode usage"

      - alert: DiskWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes[1h], 24 * 3600) < 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Disk will fill within 24h on {{ $labels.instance }}"
          description: "{{ $labels.mountpoint }} predicted to fill based on current rate"

Automatic cleanup with log rotation trigger

Pair this with a rotation policy — the Backup Rotation Script recipe covers retention patterns that plug into the cleanup below.

#!/usr/bin/env bash
set -euo pipefail

CLEANUP_THRESHOLD="${1:-90}"
LOG_DIR="${2:-/var/log/app}"
ARCHIVE_DIR="${3:-/var/log/archive}"

df -Hl | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs usage mount; do
    usage_val="${usage%\%}"
    if (( usage_val >= CLEANUP_THRESHOLD )); then
        echo "[$(date -Iseconds)] CRITICAL: $mount at $usage — triggering cleanup"

        # Compress logs older than 7 days
        find "$LOG_DIR" -name "*.log" -mtime +7 -exec gzip -9 {} \;

        # Move compressed logs to archive
        find "$LOG_DIR" -name "*.gz" -mtime +1 -exec mv {} "$ARCHIVE_DIR/" \;

        # Delete archives older than 90 days
        find "$ARCHIVE_DIR" -name "*.gz" -mtime +90 -delete

        # Clean package manager cache
        if command -v apt-get &>/dev/null; then
            apt-get clean
        elif command -v yum &>/dev/null; then
            yum clean all
        fi

        # Clean Docker if present
        if command -v docker &>/dev/null; then
            docker system prune -f --volumes
        fi

        # Report freed space
        new_usage=$(df -Hl "$mount" | awk 'NR>1 {print $5}')
        echo "[$(date -Iseconds)] Cleanup complete. $mount now at $new_usage"
    fi
done

What Works

  1. Set thresholds below 100%. Alert at 80% and take action at 90% so you’ve got time to respond.
  2. Monitor mount points, not just total disk. A small /tmp or /var/log can fill independently of the root disk.
  3. Include the filesystem in the alert. Knowing which partition is full speeds up the response.
  4. Run from cron every few minutes. A runaway log can fill a partition between hourly checks.
  5. Pair monitoring with cleanup. A high-usage alert without a cleanup plan is only half a solution — the log rotation recipe is the usual complement.
  6. Monitor inode usage separately. A disk can have free space but run out of inodes (file slots). This happens with workloads that create millions of small files — mail servers, cache directories, session storage. Check inodes with df -i and alert on the same thresholds.
  7. Use predictive alerting. Instead of alerting on current usage, alert on predicted usage. Prometheus predict_linear() can forecast when a disk will fill based on the rate of change. That turns a 2 AM emergency into a ticket you handle during business hours.
  8. Exclude read-only and tmpfs mounts. Filter out tmpfs, devtmpfs, overlay, and read-only filesystems from monitoring. They either can’t be cleaned or are managed by the kernel:
# Only monitor real block devices, excluding tmpfs and overlays
df -Hl -x tmpfs -x devtmpfs -x overlay -x squashfs | awk 'NR>1 && /^\/dev/{print $1, $5, $6}'
  1. Track disk usage trends. Log daily usage to a file or database. Trends reveal which mounts grow fastest and help plan capacity upgrades before alerts fire:
# Append daily usage to a CSV for trend analysis
echo "$(date -Iseconds),$(hostname),$mount,$usage" >> /var/log/disk-usage-trends.csv

Common Mistakes

  1. Using df -h without parsing care. The percent column can be empty for special filesystems; filter by /dev/ entries.
  2. Alerting too late. Waiting until 95% leaves almost no time to react.
  3. Ignoring ephemeral mounts. /tmp and docker volumes can fill fast and crash services.
  4. Sending alerts to individuals. Use a team alias or on-call rotation so vacations don’t break alerting.
  5. Not handling mail failures. If the MTA is down, the alert never arrives; log to a second channel.
  6. Not monitoring Docker storage. Docker uses /var/lib/docker which can grow rapidly with images, containers, and volumes. Monitor this path separately and schedule regular docker system prune jobs. When that partition fills up, new containers simply won’t start.
  7. Forgetting about mounted network volumes. NFS and SMB mounts can fill up on the remote server, causing local writes to hang. Monitor network mounts with shorter timeouts and alert on latency as well as usage. Use timeout with df to avoid hanging on unresponsive NFS servers:
timeout 10 df -Hl "$NFS_MOUNT" || echo "NFS mount $NFS_MOUNT is unresponsive"
  1. Using percentage thresholds on very large or very small disks. On a 100TB disk, 80% means 20TB free — plenty of space. On a 10GB disk, 80% means 2GB free — critical. Use absolute byte thresholds for small disks and percentage thresholds for large ones:
# Alert if free space is below 5GB OR usage is above 90%
avail_bytes=$(df -B1 "$mount" | awk 'NR>1 {print $4}')
if (( avail_bytes < 5368709120 )) || (( usage_val >= 90 )); then
    echo "WARNING: $mount has only $((avail_bytes / 1073741824))GB free"
fi

Frequently Asked Questions

How do I monitor a fleet of servers?

Run the script on each server via cron and send alerts to a centralized logging or alerting system. Better yet, let Ansible or your config management of choice push it everywhere. On AWS fleets, the AWS CLI Bash Scripts recipe covers the same automation pattern across instances.

Can I check disk usage of a specific directory?

Yes. Use du -sh /path to check a single directory, but for partition-level alerts use df because du doesn't detect mount point limits.

How do I avoid alert fatigue when a disk stays full?

Only alert on state changes. Keep a state file of mounts already alerted (see the state-change variant above) and notify again when the mount recovers.

Should I use percentage or absolute thresholds?

Percentage works for most disks, but on very small volumes alert on absolute free space too — 10% of a 10GB disk is 1GB, which can vanish in minutes.

Should I auto-delete files when disk is full?

Only if you've audited what gets deleted. Auto-deletion can wipe evidence you need for debugging — archiving logs or paging an operator is safer.