tai/runbooks/disk.md

service, symptoms, tags

service	symptoms	tags
disk	no space left on device, disk full, inode exhaustion, df shows 100%, du large files, write failed, cannot create file, filesystem read-only, ext4 error	disk, filesystem, storage, inodes, df, du, ext4, xfs, lvm, partition, full, space

Symptoms

• No space left on device — disk or inode exhaustion
• df -h shows a filesystem at 100% (or near 100%)
• df -i shows inode usage at 100% — file count exhausted even if byte space is free
• Filesystem remounted read-only — kernel detected errors and protected itself
• Services failing to write logs, create temp files, or open sockets

Diagnostics

Overall disk usage

df -h
df -i

df -h shows byte space; df -i shows inode usage. Both can be independently exhausted. Note which filesystem is full (/, /var, /tmp, /home, etc.).

Find the large directories

du -sh /* 2>/dev/null | sort -rh | head -20
du -sh /var/* 2>/dev/null | sort -rh | head -20
du -sh /var/log/* 2>/dev/null | sort -rh | head -20

Find large individual files

find / -xdev -type f -size +100M 2>/dev/null | sort -k5 -rn
find /var/log -type f -size +50M 2>/dev/null

Find deleted-but-open files holding space

lsof +L1 2>/dev/null | grep -v "^COMMAND"

Files deleted while a process still has them open do not free space until the process releases the file descriptor.

Inode exhaustion — find directories with many small files

find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

Filesystem errors (after a crash or read-only remount)

dmesg | grep -i 'ext4\|xfs\|btrfs\|error\|corrupt'
journalctl -k | grep -i 'filesystem\|disk\|io error'

LVM / partition layout

lsblk
pvs
vgs
lvs

Remediation

Large log files — truncate safely (do NOT rm while in use):

truncate -s 0 /var/log/<logfile>

Or configure log rotation in /etc/logrotate.d/.

Old journal logs eating space:

journalctl --disk-usage
journalctl --vacuum-size=500M
journalctl --vacuum-time=30d

Deleted-but-open files — restart the holding process to release space: Identify the PID from lsof +L1, then:

systemctl restart <service>

Inode exhaustion — remove many small files: Common culprits: PHP session files in /var/lib/php/sessions/, old apt cache, tmp dirs.

find /var/lib/php/sessions -type f -mtime +7 -delete
apt-get clean
find /tmp -type f -mtime +3 -delete

Extend LVM volume (if free extents exist in the volume group):

lvextend -l +100%FREE /dev/<vg>/<lv>
resize2fs /dev/<vg>/<lv>      # ext4
xfs_growfs /mountpoint         # xfs