A full disk brings a server to its knees: SSH refuses connections, databases stop, the system goes read-only. This guide shows how to monitor space and inodes with df, find what takes up the space with du and ncdu, unblock the trap of the deleted file still open, and reclaim space without breaking the system. For beginner to intermediate administrators. Outputs captured on AlmaLinux.
What you will learn
- Monitor space and inodes with
df - Find the largest directories with
duandncdu - Unblock a deleted file that is still open
- Reclaim space safely
See free space with df
df (disk free) sums up the usage of each mounted file system. The -h option makes sizes readable, -T adds the type, and excluding tmpfs removes the noise of virtual file systems:
df -hT -x tmpfs -x devtmpfsFilesystem Type Size Used Avail Use% Mounted on/dev/sdb2 ext4 20G 1.2G 19G 6% //dev/sdb1 vfat 99M 8.8M 90M 9% /boot/efiThe Use% column spots at a glance the volumes close to saturation. It is the first command to run when a "disk full" alert comes in.
Watch inodes: the trap of the "full" disk
A file system can show free space and still refuse to write: it has run out of inodes, the structures that describe each file. Their number is fixed at creation of the file system. Millions of small files (PHP sessions, caches, mail) can exhaust them well before space. Check with df -i:
df -i -x tmpfs -x devtmpfsFilesystem Inodes IUsed IFree IUse% Mounted on/dev/sdb2 2557440 36172 2521268 2% /If IUse% is at 100% while df -h shows space, the culprit is an inode shortage, not a lack of space. The fix is to delete the small files involved (or to recreate the file system with more inodes).
Find what takes up the space with du
df says how much; du (disk usage) says where. To find the big items of a directory, list the first level and sort:
du -h --max-depth=1 /usr | sort -h38M /usr/sbin64M /usr/bin169M /usr/lib64241M /usr/share505M /usr/lib1022M /usrThe sort -h sort (human-numeric) puts the biggest at the bottom, immediately visible. For the total of a folder alone, du -sh /var. Going down from directory to directory, you trace back to the source of the growth.
Explore interactively with ncdu
For a faster diagnosis, ncdu (NCurses Disk Usage) combines du and a navigable interface: sorted sizes, arrow-key navigation, deletion with the d key.
sudo apt install ncdusudo dnf install ncdusudo ncdu /varInteractive exploration saves precious time on a server under alert: you go down visually to the faulty folder instead of chaining du commands.
The trap of the deleted file still open
A classic and puzzling symptom: you delete a big file, but df still shows the disk full. The cause: a process keeps the file open. As long as it runs, the space is not freed, even though the file has disappeared from the tree. Find these files:
sudo lsof +L1The last column lists the (deleted) files still open, with the process and its PID. The most frequent case is an application log deleted without reloading the service that writes it.
Do not pass a mount point as an argument: contrary to what you would expect, it filters nothing. Measured on AlmaLinux 10, lsof +L1 alone returns 9 lines, lsof +L1 /tmp returns 17, and lsof +L1 / returns 1381. The argument selects processes, all of whose entries are then listed, libraries from other file systems included. To narrow down, filter the output:
sudo lsof +L1 | grep '/var/log'Three ways out, in order of preference:
-
Empty the file through its descriptor, without stopping the process. It is often the only acceptable option in production, and most guides leave it out:
Terminal window sudo ls -l /proc/<PID>/fd | grep '(deleted)'sudo sh -c ': > /proc/<PID>/fd/<FD>'The space is given back immediately, the service keeps running and writing to its descriptor.
-
Cleanly restart the service involved.
-
Kill the process, as a last resort.
The deleted file is not the only cause of a disk that fills up for no visible reason. On a machine managed by systemd, the journal also grows continuously, in /var/log/journal, and it is purged with its own commands rather than with an rm that could corrupt it.
Reclaim space for good
When cleaning is no longer enough, you have to grow the volume. The approach depends on the underlying layer, detailed in the dedicated guides:
- LVM volume:
lvextend -rgrows volume and file system online (see LVM). - ext4 partition: grow the partition then
resize2fs. - XFS partition: grow then
xfs_growfs(an XFS is not shrunk in practice: shrinking is still experimental).
To limit consumption per user or group upstream, set up disk quotas, covered in the quotas guide.
Troubleshooting
A full disk almost always has one of the causes below, and each one is confirmed by a command before any deletion. Start with df -h and df -i, which already tell a lack of space from a lack of inodes.
| Symptom | Likely cause | Fix |
|---|---|---|
Disk "full" but df -h shows space | inodes exhausted | df -i, delete the small files involved |
| Space not freed after deletion | deleted file still open | lsof +L1, then empty via /proc/<PID>/fd/<FD> or restart the service |
lsof +L1 <mount-point> returns more lines than lsof +L1 alone | The argument selects processes, not a file system | Filter the output with grep, the argument narrows nothing |
| Service restarted, space still not given back | The descriptor is inherited by child processes | lsof +L1 lists parent and children with the same inode: killing the parent alone is not enough |
df says 100% but root still writes | ext4 reserve of 5% kept for root | tune2fs -l <dev> | grep -i reserved; tune2fs -m 1 to lower it |
du -sh clearly underestimates the space used | Unreadable directories, errors swallowed | Run again with sudo; without it, the Permission denied go by silently |
/ fills up for no visible reason | systemd journals or application logs | journalctl --vacuum-size, check /var/log |
/boot full after updates | old kernels piled up | purge obsolete kernels according to the distribution |
Put it into practice
Facing a real full disk, theory is no longer enough: you must find the culprit fast and without breaking legitimate data. The lab confronts you with a saturated file system that you diagnose with df and du, before deleting the stray files and bringing usage back under the alert threshold.
Key points
df -hshows space;df -ithe inodes (a disk can be "full" without lacking space).du -h --max-depth=1 | sort -hfinds the big directories;ncduexplores interactively.- A deleted file still open does not free space:
lsof +L1, then empty it via/proc/<PID>/fd/<FD>or restart the service. - systemd journals are purged with
journalctl --vacuum-size/--vacuum-time. - To grow for good:
lvextend -r(LVM),resize2fs(ext4) orxfs_growfs(XFS). - Quotas limit consumption per user upstream.
Knowledge check
Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Use NFS on Linux
A new full quiz with random questions
Work again only on the questions you missed