Analyse disk performance on Linux

A service is slow and you suspect the disk? Before concluding, you have to measure. This guide shows how to assess the three indicators that really matter (throughput, IOPS, latency) with fio and iostat, check the health of the hardware with smartctl, and above all interpret the results without falling into the trap of the cache or of a test that is too short. It is aimed at intermediate to advanced administrators on Linux (Debian, Ubuntu, RHEL).

What you will learn

• Tell throughput, IOPS and latency apart, and know which one to look at
• Measure a disk's real performance with fio
• Observe live activity with iostat and iotop
• Avoid hasty conclusions (cache, test too short, misleading %util)

Prerequisites

• A Linux machine with a sudo account
• The fio, sysstat (for iostat) and smartmontools tools installed
• Understand the notion of file system and mounting

Debian / Ubuntu

sudo apt install fio sysstat smartmontools

RHEL / Fedora

sudo dnf install fio sysstat smartmontools

Throughput, IOPS, latency: the three metrics

The first mistake is to talk about a disk's "speed" as a single figure. Three different quantities describe storage, and a slow service almost never suffers from all three at once.

Metric	What it measures	When it matters
Throughput (MB/s)	volume transferred per second	large files, backups, video
IOPS	number of operations per second	databases, many small files
Latency (await, ms or µs)	response time of one operation	user experience, interactive applications

Swipe to see more

The type of disk radically changes the orders of magnitude. A hard disk (HDD) tops out around 100-200 IOPS because of the mechanical head; a SATA SSD reaches tens of thousands of IOPS; an NVMe exceeds a million. Comparing the latency of an NVMe (a few tens of µs) with that of an HDD (several ms) makes no sense: they are different worlds.

The right reflex

A slow database is almost always a problem of IOPS or latency, not throughput. Measuring the sequential throughput of an SSD to diagnose a slow database is a waste of time: look at the random IOPS and the await.

Check health before measuring

No point benchmarking a dying disk. smartctl (smartmontools package) reads the S.M.A.R.T. data the disk keeps about itself:

sudo smartctl -a /dev/sda

Three attributes warn of an approaching end of life:

• Reallocated_Sector_Ct: faulty sectors already replaced.
• Current_Pending_Sector: unstable sectors waiting for reallocation.
• Wear_Leveling_Count (SSD): wear of the memory cells.

A counter that climbs is a replacement signal. Also run a non-destructive self-test:

sudo smartctl -t short /dev/sda     # short test (a few minutes)
sudo smartctl -l selftest /dev/sda  # read the result

Observe real activity with iostat

iostat (sysstat package) shows what the disks are doing live. It is the tool to see whether a disk is saturated while the problem is happening:

iostat -dx 1

The -d option targets disks, -x adds the extended columns, and 1 refreshes every second. Here is an excerpt of the key columns (the full output has more):

Device   r/s     rkB/s    r_await   w/s     wkB/s    w_await   aqu-sz   %util
dm-0     55.93   1167.49   0.69      30.58   493.59    1.29      0.08     1.55

What to read:

• r/s and w/s: read and write operations per second (the real IOPS).
• rkB/s and wkB/s: read and write throughput.
• r_await and w_await: the average latency in milliseconds, split by direction. It is the most telling indicator of a bottleneck.
• aqu-sz: average queue size; when high, it reveals an overwhelmed disk.
• %util: percentage of busy time.

svctm and %util hardly mean anything any more

Old guides quote svctm and read %util at 100% as "disk saturated". In sysstat 12+, the svctm column is gone because it was misleading. And on an SSD or NVMe, %util close to 100% does not mean saturation: these disks process several requests in parallel, so "busy" does not mean "overwhelmed". Rely on await and aqu-sz, not on %util alone.

To know which process generates the I/O, iotop lists the culprits in real time:

sudo iotop -oP

The -o option hides idle processes, -P groups by process. That is how you spot a log service or a backup hammering the disk.

On RHEL and AlmaLinux 10, the package is called `iotop-c`

dnf install iotop answers No matching Packages to list on AlmaLinux 10: the historical tool, written in Python, is gone. Its C rewrite, iotop-c (1.26), is in the BaseOS repository and installs a command named iotop, with the same options:

sudo dnf install -y iotop-c
sudo iotop -oP

Without any extra package, pidstat -d 2 (sysstat package) answers the same question, who writes, giving per process the kilobytes read and written per second.

Benchmark with fio

iostat observes the real load; fio (Flexible I/O tester) provokes it in a controlled way to measure what the disk is capable of. It is the reference tool for storage benchmarking.

Test on a file, never on a disk in service

fio can write directly to a raw device (--filename=/dev/sdX), which destroys the data. For a safe test, always point to a file in a mounted directory (--directory), as below.

To measure random 4 KB IOPS, the critical case for databases:

fio --name=randread --directory=/var/tmp --rw=randread \
    --bs=4k --size=1G --runtime=30 --time_based \
    --ioengine=libaio --direct=1 --group_reporting

read: IOPS=22.3k, BW=87.2MiB/s (91.5MB/s)(2617MiB/30001msec)
   lat (usec): min=21, max=3954, avg=44.07, stdev=44.74

The line gives everything: 22,300 IOPS, a throughput of 87.2 MiB/s, and an average latency of 44 µs, measured with fio 3.36 on the NVMe of a workstation, with no virtualisation. To measure sequential throughput (large transfers), change the pattern and the block size:

fio --name=seqread --directory=/var/tmp --rw=read \
    --bs=1M --size=2G --runtime=30 --time_based \
    --ioengine=libaio --direct=1 --group_reporting

read: IOPS=2850, BW=2850MiB/s (2989MB/s)(83.5GiB/30001msec)

The decisive parameters: --rw (randread, randwrite, read, write, randrw), --bs (4k for random, 1M for sequential), and above all --direct=1, explained right after.

Avoid hasty conclusions

This is where most analyses go wrong. Four classic traps, each of which can turn a measurement into a wrong conclusion.

The cache distorts everything. Without --direct=1, you mostly measure RAM, not the disk: Linux serves reads from its page cache. A naive dd shows flattering GB/s that reflect nothing:

# Misleading: reads/writes in the cache
dd if=/dev/zero of=/var/tmp/test bs=1M count=1024
# More honest: forces the physical write
dd if=/dev/zero of=/var/tmp/test bs=1M count=1024 oflag=direct

With fio, --direct=1 bypasses the cache: it is non-negotiable to measure the disk itself.

In a virtual machine, `--direct=1` only bypasses the guest's cache

It is the limit to know before publishing a figure, and it concerns most readers, since most measure in a VM. O_DIRECT asks the guest kernel not to go through its page cache. It has no hold on the hypervisor's cache, which keeps serving blocks from the host's RAM.

Measured on a KVM VM, fio --direct=1 --iodepth=1 in 4 k random reads:

read: IOPS=73.2k, BW=286MiB/s
clat (usec): min=7, max=1688, avg=13.40

73,200 IOPS at 13.4 µs average latency, with a single request in flight. No physical medium reaches that: the NVMe measured above on bare metal gives 44 µs on average at iodepth=1, and most sit between 40 and 100 µs. The figure therefore describes the host's RAM, not the disk.

Two practical consequences. First, --direct=1 stays right on bare metal, and the advice fully applies there. Second, in a VM, an average latency below 20 µs should alert you: it is the signature of a measurement of the hypervisor's cache, not of the storage. In that case, the only honest measurement is taken from the host, or by having the virtual disk configured with cache=none.

A test that is too short lies. A few seconds measure a peak, not sustained behaviour. An SSD with a good start can collapse once its SLC cache is full. Use --runtime=30 --time_based at the very least.

%util at 100% is not proof on SSD/NVMe, as seen above: look at the latency.

Measuring with no load makes no sense. The only test that counts is under a load close to reality: same type of access (random or sequential), same block size as your application.

Reproduce this guide

Everything can be tested without risk on a file: fio on /var/tmp touches no data. Run the two fio commands above on your machine and compare your random IOPS (4k) with your sequential throughput (1M): the gap between the two says everything about the nature of your storage.

Troubleshooting

A disk performance diagnosis rarely picks the wrong tool, it picks the wrong reading. Before any action, cross-check the latency (await) with the S.M.A.R.T. health state and the type of disk:

Symptom	Likely cause	Lead
High await (> 20 ms on SSD)	disk saturated or failing	check smartctl, spread the I/O load
%util at 100% but app slow	false positive on parallel SSD/NVMe	look at await and aqu-sz, not %util
Very low random IOPS	HDD, or expected cache disabled	compare with the disk type, check --direct
Unrealistic fio throughput (GB/s)	cache not bypassed	add --direct=1
A single process saturates the disk	badly tuned service (logs, backup)	identify it via iotop -oP (iotop-c on RHEL 10) or pidstat -d, adjust its pace

Swipe to see more

Hardware to practise

To measure real NVMe performance with fio, the Samsung 990 Pro is a reference (high sequential throughput, sustained random IOPS), ideal to reproduce the tests of this guide.

Amazon affiliate link. Buying through this link supports the site at no extra cost to you.

Put it into practice

After measurement comes optimisation: a heavily read volume wastes I/O writing access dates. In this lab, you add the noatime option on a read-heavy mount, enable it live and write it into /etc/fstab so that the performance setting stays in place after a reboot.

Guided lab Practice

Learn how to do it

Tune a mount for performance with noatime (persistently)

$ dsoxlab run l2-storage-performance

Environment: VM·Duration: ~20 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start l2-storage-performance checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/linux-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run l2-storage-performance
dsoxlab check l2-storage-performance

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

• Three distinct metrics: throughput (MB/s), IOPS, latency (await). A slow database = IOPS/latency, not throughput.
• fio measures what the disk is capable of; iostat observes the real load.
• --direct=1 is essential: without it, you measure the cache, not the disk.
• On SSD/NVMe, %util at 100% proves nothing; svctm is gone from sysstat 12.
• Check S.M.A.R.T. health before benchmarking, and test on a file, never on a disk in service.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →