df -h shows the space used per filesystem, df -i shows the inodes; when either reaches 100%, Linux returns No space left on device even if room remains on the other side. The diagnosis always starts with these two commands. This guide is written for the administrator who must free up space on a production server without breaking anything: find what consumes it with du and find, recover the space of deleted files that are still open with lsof, understand ext4's reserved blocks and contain the systemd journal. It ends with a cleanup procedure in the right order.
What you will learn
- Tell apart space saturation and inode saturation
- Identify the largest directories and files with
du - Detect deleted files that are still open with
lsof - Diagnose inode exhaustion (
df -i) - Free up space safely in production
- Understand the role of reserved blocks on ext4
In what context?
Checking disk space comes up constantly in administration:
- A service refuses to start or an application crashes with
No space left on device: the reflex is to checkdf -h - After a failed log rotation, /var/log grew insidiously and fills the partition
- A container or a CI build writes temporary files that are never cleaned up
dfshows available space but file creation fails: it is an inode problem- You must justify resizing a partition or a logical volume to the infrastructure team
Check the available space with df
df (disk free) queries each mounted filesystem and reports the usage as seen by the kernel. It is always the first command to run, because it reasons by mount point: a server can have / at 40% and /var at 100%, and only df shows it. The -h option (human readable) converts blocks into readable units; without it, sizes are shown in 1-kilobyte blocks.
Space per filesystem
Without arguments, df lists all mounted filesystems. The tmpfs lines are filesystems in RAM, which disappear at reboot and consume no disk space.
df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/ubuntu--vg-ubuntu--lv 914G 616G 260G 71% /tmpfs 24G 1.2M 24G 1% /run/dev/sda2 2.0G 130M 1.7G 8% /boot| Column | Meaning |
|---|---|
Size | Total size of the filesystem |
Used | Space used |
Avail | Space really available (excludes reserved blocks) |
Use% | Usage percentage |
Mounted on | Mount point |
Filter on a mount point
When given a path, df only shows the filesystem that contains that path. Useful to clear up a doubt: if df -h /var returns the same line as df -h /, then /var is not on a separate partition.
df -h /vardf -h /tmpSpace as a percentage for a script
For an automated alert, extract the only value that matters. NR==2 skips the header line and $5 designates the Use% column.
df -h / | awk 'NR==2 {print $5}'71%Usable in an alert script:
usage=$(df -h / | awk 'NR==2 {gsub(/%/,""); print $5}')if [[ "$usage" -gt 90 ]]; then echo "ALERT: / is at ${usage}%"fiCheck the inodes with df -i
A filesystem manages two independent resources: data space, expressed in bytes, and inodes, which count files. One can be exhausted while the other is almost empty, and the error message returned by the kernel is identical in both cases. That is why a complete diagnosis requires df -h then df -i.
Why inodes matter
Each file, directories included, uses an inode: a metadata structure that stores the permissions, the timestamps and the pointers to the data blocks. On ext4, their total number is fixed at creation of the filesystem. When all inodes are used up, creating a new file is impossible even if the disk space is not full.
df -i /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/ubuntu--vg-ubuntu--lv 60850176 32625792 28224384 54% /| Column | Meaning |
|---|---|
Inodes | Total number of inodes of the filesystem |
IUsed | Inodes used |
IFree | Inodes available |
IUse% | Inode usage percentage |
When to suspect an inode problem
Three signals should steer the diagnosis towards inodes rather than space. The first one is decisive: a write that fails while df -h shows free gigabytes cannot be explained any other way.
df -hshows available space buttouch /tmp/testreturnsNo space left on device- A directory contains millions of small files (cache, PHP sessions, queued mail)
IUse%exceeds 90% indf -i
Find the directories with the most files
There is no du equivalent for counting inodes: you must count files directory by directory. The command below walks the whole root filesystem and ranks directories by number of direct files. It is slow to run on a large disk, -xdev at least limiting the walk to the current filesystem.
find / -xdev -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 -type f | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head -10Identify what consumes the space with du
Where df queries the kernel, du (disk usage) actually walks the tree and adds up file sizes. This difference in method explains two behaviours: du is much slower, and it can total noticeably less than df when deleted files remain open. Run it with sudo so as not to miss directories your account cannot read, and redirect errors to /dev/null to keep a usable output.
The 10 largest directories
The sort -rh sort ranks by decreasing size while understanding the G, M and K suffixes produced by -h. The first line is always the total of the starting point, here /.
sudo du -ahx / 2>/dev/null | sort -rh | head -10188G /166G /home87G /home/bob/Projets35G /home/bob/.cache29G /home/bob/Projets/lab| Option | Role |
|---|---|
-a | Also shows files (not only directories) |
-h | Readable sizes (G, M, K) |
-x | Stays on the same filesystem (does not cross mounts) |
Do not compare this total with the df -h shown above: the two outputs come from different machines. The gap between df and du on one and the same machine is a signal to read, and it is covered below; between two machines, it means nothing.
Target a specific directory
Once the culprit is located at the global level, go down one level. The -s (summarize) option only shows a total per argument instead of detailing the whole tree, which makes the output immediately readable.
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -5633M /var/log/journal13M /var/log/pods13M /var/log/crowdsec_api.log2.5M /var/log/harbor1.5M /var/log/libvirtSummary per top-level directory
This is the command to run first when you discover a machine: it fits in about ten lines and immediately points to the branch to explore.
sudo du -sh /* 2>/dev/null | sort -rhIt gives an immediate view of the overall distribution.
Find the large files with find
du points to directories, find points to files. Switching is useful when a single file explains the saturation: a forgotten disk image, a database dump, a log never rotated. Three criteria cover most searches: the size (-size), the modification date (-mtime) and the last access date (-atime).
Files larger than 100 MB
-xdev keeps find from going down into other mounted filesystems, which avoids walking a network share or a backup disk. The sort applies to the fifth column of ls -lh, the size.
sudo find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -10Recently modified and large files
Crossing size and freshness answers another question: what is growing right now. The sign of -mtime reverses the criterion, -7 meaning the last 7 days and +7 anything older.
sudo find /var -type f -size +10M -mtime -7 -exec ls -lh {} \; 2>/dev/nullIt targets files over 10 MB modified in the last 7 days under /var, useful to find a fast-growing log.
Old temporary files
-atime relies on the last access date, more relevant than the modification date to spot what is no longer used. This information is only reliable if the filesystem is not mounted with the noatime option, common on servers to save writes.
find /tmp -type f -atime +30 -exec ls -lh {} \; 2>/dev/nullFiles in /tmp not accessed for more than 30 days, candidates for cleanup.
Detect deleted files that are still open
A file deleted with rm but still open by a process keeps using disk space. It is the most frequent cause of "ghost" space: du sees nothing, df shows the filesystem full.
sudo lsof +L1COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAMEnginx 1234 www 4w REG 253,0 2.1G 0 131074 /var/log/nginx/access.log (deleted)The kernel keeps the blocks as long as a descriptor remains open on this file. There are two ways to release them, and the first avoids any service interruption: empty the file through its descriptor, which /proc/<PID>/fd/ exposes.
# 1. without interrupting the service: truncate the file that is still opensudo ls -l /proc/1234/fd | grep '(deleted)'sudo sh -c ': > /proc/1234/fd/4'
# 2. otherwise, restarting the service closes the descriptorsudo systemctl restart nginxThe first form is often the only acceptable one in production, where you do not restart a service to recover disk space. The detailed procedure and its pitfalls are in Monitor disk capacity and quotas.
In both cases, df -h reflects the recovered space immediately: the kernel releases the blocks as soon as the last descriptor closes or the file is truncated.
Understand reserved blocks (ext4)
On ext4, 5% of the space is reserved by default for the root user. This explains the difference between Size, Used and Avail in df:
sudo tune2fs -l /dev/mapper/ubuntu--vg-ubuntu--lv | grep -i reservedReserved block count: 10002749Reserved blocks uid: 0 (user root)Reserved blocks gid: 0 (group root)This reserved space lets root keep writing (system logs, critical processes) even when a regular user sees No space left.
Reduce the reserved blocks
On a pure data volume (not /), you can reduce the reserved percentage:
sudo tune2fs -m 1 /dev/mapper/data-lvIt changes the reserve from 5% to 1%. Do not do this on /: system processes need this margin.
Check the space used by the systemd journal
The systemd journal can take up several hundred MB:
journalctl --disk-usageArchived and active journals take up 632.5M in the file system.To limit it to 200 MB:
sudo journalctl --vacuum-size=200MTo keep only the last 7 days:
sudo journalctl --vacuum-time=7dMake the limit persistent in /etc/systemd/journald.conf:
[Journal]SystemMaxUse=200MThen reload:
sudo systemctl restart systemd-journaldCleanup procedure in production
On a running server, the order of operations matters as much as the commands. You diagnose before deleting, you start with reversible candidates, caches, old journals and orphaned packages, and you only touch application data as a last resort. Deletion on Linux being irreversible, each step below is limited to targets the system knows how to regenerate.
-
Diagnose: space or inodes?
Terminal window df -h /df -i / -
Locate: which directories consume the most?
Terminal window sudo du -sh /var/* /tmp/* /home/* 2>/dev/null | sort -rh | head -10 -
Check the deleted open files
Terminal window sudo lsof +L1 -
Clean the safe candidates
Terminal window # Old journalssudo journalctl --vacuum-time=7d# Package cachesudo apt clean # Debian/Ubuntusudo dnf clean all # RHEL/Fedora# Old temporary filessudo find /tmp -type f -atime +30 -delete# Old kernels (Debian/Ubuntu)sudo apt autoremove --purge -
Check the result
Terminal window df -h /
Troubleshooting
The first two lines of the table cover the most confusing situations, those where the commands seem to contradict each other. Remember the reading rule: when df and du disagree, the difference is almost always taken up by deleted files that are still open.
| Symptom | Probable cause | Solution |
|---|---|---|
No space left on device but df -h shows free space | Inodes exhausted | df -i to confirm, delete useless small files |
df full but du / shows much less | Deleted files still open | lsof +L1, then truncate through /proc/<PID>/fd/<FD>; restart the service only if that is impossible |
/var fills up very quickly | Unrotated logs or systemd journal without a limit | Configure logrotate and SystemMaxUse in journald.conf |
Avail smaller than Size - Used | Blocks reserved for root (5% by default on ext4) | Normal; reduce with tune2fs -m if it is a data volume |
| Alert script does not detect the saturation | Wrong mount point checked | Check /var, /tmp, /home separately |
| Partition full after a Docker build | Images and layers not cleaned up | docker system prune or podman system prune |
Key points
- Two causes of
No space left on device: full space (df -h) or exhausted inodes (df -i); always check both du -shidentifies large directories;find -sizetargets individual large files- Deleted files that are still open (
lsof +L1) are the most frequent cause of ghost space - Reserved blocks for root (5% on ext4) explain the difference between total and available space
- The systemd journal can be limited with
SystemMaxUsein/etc/systemd/journald.conf - On XFS, inodes are dynamic and exhaustion is much rarer than with ext4, without being impossible: their share is capped at 25% of the space by default
Knowledge check
Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Local storage on Linux: the complete path
A new full quiz with random questions
Work again only on the questions you missed