107 lines
2.6 KiB
Markdown
107 lines
2.6 KiB
Markdown
---
|
|
service: disk
|
|
symptoms: no space left on device, disk full, inode exhaustion, df shows 100%, du large files, write failed, cannot create file, filesystem read-only, ext4 error
|
|
tags: disk, filesystem, storage, inodes, df, du, ext4, xfs, lvm, partition, full, space
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- `No space left on device` — disk or inode exhaustion
|
|
- `df -h` shows a filesystem at 100% (or near 100%)
|
|
- `df -i` shows inode usage at 100% — file count exhausted even if byte space is free
|
|
- Filesystem remounted read-only — kernel detected errors and protected itself
|
|
- Services failing to write logs, create temp files, or open sockets
|
|
|
|
## Diagnostics
|
|
|
|
### Overall disk usage
|
|
|
|
```
|
|
df -h
|
|
df -i
|
|
```
|
|
|
|
`df -h` shows byte space; `df -i` shows inode usage. Both can be independently exhausted.
|
|
Note which filesystem is full (`/`, `/var`, `/tmp`, `/home`, etc.).
|
|
|
|
### Find the large directories
|
|
|
|
```
|
|
du -sh /* 2>/dev/null | sort -rh | head -20
|
|
du -sh /var/* 2>/dev/null | sort -rh | head -20
|
|
du -sh /var/log/* 2>/dev/null | sort -rh | head -20
|
|
```
|
|
|
|
### Find large individual files
|
|
|
|
```
|
|
find / -xdev -type f -size +100M 2>/dev/null | sort -k5 -rn
|
|
find /var/log -type f -size +50M 2>/dev/null
|
|
```
|
|
|
|
### Find deleted-but-open files holding space
|
|
|
|
```
|
|
lsof +L1 2>/dev/null | grep -v "^COMMAND"
|
|
```
|
|
|
|
Files deleted while a process still has them open do not free space until the process releases the file descriptor.
|
|
|
|
### Inode exhaustion — find directories with many small files
|
|
|
|
```
|
|
find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20
|
|
```
|
|
|
|
### Filesystem errors (after a crash or read-only remount)
|
|
|
|
```
|
|
dmesg | grep -i 'ext4\|xfs\|btrfs\|error\|corrupt'
|
|
journalctl -k | grep -i 'filesystem\|disk\|io error'
|
|
```
|
|
|
|
### LVM / partition layout
|
|
|
|
```
|
|
lsblk
|
|
pvs
|
|
vgs
|
|
lvs
|
|
```
|
|
|
|
## Remediation
|
|
|
|
**Large log files — truncate safely (do NOT rm while in use):**
|
|
```
|
|
truncate -s 0 /var/log/<logfile>
|
|
```
|
|
Or configure log rotation in `/etc/logrotate.d/`.
|
|
|
|
**Old journal logs eating space:**
|
|
```
|
|
journalctl --disk-usage
|
|
journalctl --vacuum-size=500M
|
|
journalctl --vacuum-time=30d
|
|
```
|
|
|
|
**Deleted-but-open files — restart the holding process to release space:**
|
|
Identify the PID from `lsof +L1`, then:
|
|
```
|
|
systemctl restart <service>
|
|
```
|
|
|
|
**Inode exhaustion — remove many small files:**
|
|
Common culprits: PHP session files in `/var/lib/php/sessions/`, old apt cache, tmp dirs.
|
|
```
|
|
find /var/lib/php/sessions -type f -mtime +7 -delete
|
|
apt-get clean
|
|
find /tmp -type f -mtime +3 -delete
|
|
```
|
|
|
|
**Extend LVM volume (if free extents exist in the volume group):**
|
|
```
|
|
lvextend -l +100%FREE /dev/<vg>/<lv>
|
|
resize2fs /dev/<vg>/<lv> # ext4
|
|
xfs_growfs /mountpoint # xfs
|
|
```
|