Monitor Disk Usage

Overview

Disk space is a silent killer of production services. When a log, cache, or database fills the disk, applications crash, writes fail, and recovery becomes urgent. A simple Bash script that checks mount points and alerts when usage crosses a threshold gives you early warning and can trigger cleanup before the situation becomes critical.

When to Use

Use this resource when:

• You want lightweight monitoring without installing a full agent.
• You need to alert via email, Slack, or a log file when disk usage is high.
• You run containers, VMs, or bare-metal servers with limited disk.
• You want to trigger automatic cleanup when usage crosses a threshold.

Solution

Disk usage monitoring script

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD="${1:-80}"
EMAIL="${2:-admin@example.com}"

# Check all local mount points
df -Hl | awk 'NR>1 && /^\/dev/{print $1, $5, $6}' | while read -r fs usage mount; do
    usage_val="${usage%%}"
    if (( usage_val >= THRESHOLD )); then
        echo "WARNING: $mount ($fs) is at $usage"
        # Send alert (example with mail)
        echo "Disk usage on $mount is $usage" | mail -s "Disk alert: $mount" "$EMAIL"
    else
        echo "OK: $mount is at $usage"
    fi
done

# Optional: trigger cleanup for specific mounts
# if (( usage_val >= 90 )); then
#     /usr/local/bin/cleanup-logs.sh
# fi

Explanation

The script uses df -Hl to list local filesystems and their usage percentages. awk filters out the header and non-device entries. For each mount point, it strips the percent sign, compares the numeric value to the threshold, and prints a warning or OK message. If the threshold is exceeded, it sends an email alert. The cleanup block is commented out because automatic deletion should be carefully reviewed before enabling.

Variants

Alert channel	Tool	Best for
Email	mail	Simple servers with local MTA
Slack	webhook curl	Teams already using Slack
PagerDuty	event API	Production on-call escalation
File log	redirect to syslog	Centralized log aggregation

What Works

1. Set thresholds below 100%. Alert at 80% and take action at 90% so you have time to respond.
2. Monitor mount points, not just total disk. A small /tmp or /var/log can fill independently of the root disk.
3. Include the filesystem in the alert. Knowing which partition is full speeds up the response.
4. Run from cron every few minutes. Disk usage can grow quickly during incidents.
5. Pair monitoring with cleanup. A high-usage alert without a cleanup plan is only half a solution.

Common Mistakes

1. Using df -h without parsing care. The percent column can be empty for special filesystems; filter by /dev/ entries.
2. Alerting too late. Waiting until 95% leaves almost no time to react.
3. Ignoring ephemeral mounts. /tmp and docker volumes can fill fast and crash services.
4. Sending alerts to individuals. Use a team alias or on-call rotation so vacations do not break alerting.
5. Not handling mail failures. If the MTA is down, the alert never arrives; log to a second channel.

Additional Best Practices

• For a deeper guide, see Backup Rotation Script.

1. Monitor inode usage separately. A disk can have free space but run out of inodes (file slots). This happens with workloads that create millions of small files — mail servers, cache directories, session storage. Check inodes with df -i and alert on the same thresholds.

2. Use predictive alerting. Instead of alerting on current usage, alert on predicted usage. Prometheus predict_linear() can forecast when a disk will fill based on the rate of change. This gives you hours or days of warning instead of minutes.

3. Exclude read-only and tmpfs mounts. Filter out tmpfs, devtmpfs, overlay, and read-only filesystems from monitoring. They either cannot be cleaned or are managed by the kernel:

# Only monitor real block devices, excluding tmpfs and overlays
df -Hl -x tmpfs -x devtmpfs -x overlay -x squashfs | awk 'NR>1 && /^\/dev/{print $1, $5, $6}'

4. Track disk usage trends. Log daily usage to a file or database. Trends reveal which mounts grow fastest and help plan capacity upgrades before alerts fire:

# Append daily usage to a CSV for trend analysis
echo "$(date -Iseconds),$(hostname),$mount,$usage" >> /var/log/disk-usage-trends.csv

Additional Common Mistakes

1. Not monitoring Docker storage. Docker uses /var/lib/docker which can grow rapidly with images, containers, and volumes. Monitor this path separately and schedule regular docker system prune jobs. A full Docker storage partition prevents new containers from starting.

2. Forgetting about mounted network volumes. NFS and SMB mounts can fill up on the remote server, causing local writes to hang. Monitor network mounts with shorter timeouts and alert on latency as well as usage. Use timeout with df to avoid hanging on unresponsive NFS servers:

timeout 10 df -Hl "$NFS_MOUNT" || echo "NFS mount $NFS_MOUNT is unresponsive"

3. Using percentage thresholds on very large or very small disks. On a 100TB disk, 80% means 20TB free — plenty of space. On a 10GB disk, 80% means 2GB free — critical. Use absolute byte thresholds for small disks and percentage thresholds for large ones:

# Alert if free space is below 5GB OR usage is above 90%
avail_bytes=$(df -B1 "$mount" | awk 'NR>1 {print $4}')
if (( avail_bytes < 5368709120 )) || (( usage_val >= 90 )); then
    echo "WARNING: $mount has only $((avail_bytes / 1073741824))GB free"
fi