Diagnose memory on Linux

On Linux, memory is not what a raw percentage seems to say. A monitoring tool that shows "90 % of RAM used" often computes (total - free) / total: this figure includes the cache, reclaimable at any time, and triggers panic for nothing. The used column of free, for its part, is total - available with procps-ng 4.0.4: it leaves the reclaimable cache out. The skill to acquire in memory diagnosis is to read the right fields (available rather than MemFree, used distinct from buff/cache), to identify the processes that really consume, and to anticipate memory pressure before the OOM killer kills a critical service. This guide walks the full loop: overview, detail, processes, time, action.

What you will learn

• Tell apart used, free, available, buff/cache and swap in free -h
• Read the essential fields of /proc/meminfo without drowning in its sixty lines
• Identify the processes that really use RAM with ps and top/htop
• Observe the trend with vmstat and sar -r
• Act: swap file, swappiness, cache purge
• Anticipate the OOM killer and react when it strikes

In which context?

This diagnosis serves in three frequent scenarios, from the incident already there to the sizing planned ahead:

• A service has just been killed, the log mentions Out of memory: Killed process. You must find out who was consuming, why, and whether the machine is undersized.
• The server slows down for no obvious reason, swap peak, memory contention, a process that leaks progressively.
• Preparing a sizing, before adding a service, check the available margin and the swap posture.

Both certifications assess it. The LFCS asks to "Monitor and troubleshoot system performance and services" (Essential Commands domain) and to "Configure and manage swap space" (Storage domain). The RHCSA (EX200, RHEL 10) asks to "Identify CPU/memory intensive processes and kill processes" and to add "swap to a system non-destructively".

Prerequisites

• A terminal with sudo rights
• The basics of Linux processes
• Install htop and sysstat (for sar) if missing, then enable the collection of sar, disabled by default on Debian and Ubuntu (ENABLED="false" in /etc/default/sysstat, read on Debian 13 and Ubuntu 24.04):

Debian / Ubuntu

sudo apt update
sudo apt install -y htop sysstat
sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat

RHEL / Fedora / Rocky

sudo dnf install -y epel-release   # htop comes from EPEL on RHEL 10
sudo dnf install -y htop sysstat
sudo systemctl enable --now sysstat

The memory model in 5 minutes

Before reading the commands, five concepts are enough to read every output of this guide without misinterpretation.

Physical RAM. The main memory of the machine, a strict physical limit that no setting pushes back and every process shares.

Virtual memory. Each process has its own virtual address space, mapped to RAM by the kernel. Unused pages can move to disk (swap).

Buffers and cache. Linux keeps in RAM what was recently read from or written to disk. This memory is immediately reclaimable as soon as a process needs it. The modern free (procps-ng) puts it in its own buff/cache column and counts it in available. With procps-ng 4.0.4, the used column is computed as total - available: measured on Debian 13, Ubuntu 24.04 and AlmaLinux 10, the value of free -k lands exactly on MemTotal - MemAvailable. In other words, used measures the RAM that cannot be reclaimed, not the cache. Do not confuse the two.

Swap. Disk extension of memory. When RAM saturates, little-used pages are written there to free room. Disk access is 100 to 1000× slower than RAM, a system that swaps a lot slows down heavily.

OOM killer. When RAM + swap are saturated, the kernel kills a process to free memory and avoid a total freeze. The choice is driven by the /proc/<pid>/oom_score score.

The full-RAM myth

A healthy Linux machine uses almost all its RAM all the time, the cache fills the free space to speed up I/O. A server with lots of "free" RAM wastes performance. But mind what "full" means: the "don't worry, it's cache" myth holds for a figure computed as (total - free) / total, or a dashboard that files buff/cache under "used". It does not hold for the used column of free, which already leaves the reclaimable cache out: a high used is real occupation by processes. The only reliable health metric remains available, completed by the swap activity.

Overview with free

The first command, to read from left to right. With procps-ng 4.0.4, used and available complement each other: their sum gives total.

free -h

Typical output:

               total        used        free      shared  buff/cache   available
Mem:            31Gi       2.6Gi        25Gi        13Mi       3.9Gi        28Gi
Swap:          8.0Gi          0B       8.0Gi

The columns that matter:

Column	Meaning	Watch it?
total	Physical RAM	Constant, serves as reference
used	Non-reclaimable memory (total - available with procps-ng 4.0.4)	High = real pressure, correlate with available and swap
free	Memory not used at all	Often low, it is normal
buff/cache	I/O caches, reclaimable on demand	High = the system makes use of RAM
available	What can be given to a new process	The real health metric
Swap total/used	Configured / used swap area	A rising used = memory pressure

Swipe to see more

Useful free options

• free -h, readable units (Ki/Mi/Gi)
• free -s 2, refreshes every 2 seconds (live follow-up)
• free -t, adds a Total line combining RAM + swap

Detail through /proc/meminfo

free reads its figures from /proc/meminfo. This file exposes ~60 fields; 7 are enough for a routine diagnosis.

grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Dirty|Writeback):' /proc/meminfo

Typical output:

MemTotal:       32602704 kB
MemFree:        26288324 kB
MemAvailable:   29867396 kB
Buffers:          511084 kB
Cached:          3327372 kB
SwapTotal:       8388604 kB
SwapFree:        8388604 kB
Dirty:               252 kB
Writeback:             0 kB

The fields to remember:

• MemAvailable, what the kernel estimates can be allocated to a new process, taking the reclaimable caches into account. This is the value that counts, not MemFree.
• Buffers + Cached, reclaimable memory. A sudden drop signals strong demand.
• SwapFree, if close to zero, the OOM killer is near.
• Dirty, modified pages waiting to be written to disk. A persistently high value = I/O bottleneck.

Identify the hungry processes

With ps

Sort by RAM consumption, with the ten largest first; the minus sign reverses the order:

ps aux --sort=-%mem | head -n 11

Key columns:

Column	Meaning
USER	Owner of the process
%MEM	% of physical RAM used
VSZ	Size of the virtual space (in Ki), can be misleading, includes mapped pages never loaded
RSS	Resident Set Size, physical memory actually occupied (in Ki)
COMMAND	Command that started the process

Swipe to see more

Prefer RSS to VSZ to know what a process really consumes.

With top / htop

top gives a real-time view, sorted here by memory with the -o option:

top -o %MEM

In the interface, M sorts by memory, c shows the full command, f lets you choose the columns displayed.

htop is more readable and intuitive, with colour bars:

htop

The top bars show RAM and swap; the MEM% column sorts with a single click or with F6.

Understand shared memory with smem

ps double-counts the memory shared between several processes. smem exposes the Proportional Set Size (PSS), the share really attributable to each process. It installs with apt install smem on Debian and Ubuntu, but does not exist for RHEL 10, not even in EPEL (checked on AlmaLinux 10); the Pss field of /proc/<pid>/smaps_rollup then gives the same measure, process by process:

sudo smem -rs pss | head -n 10

# Without smem: PSS of one process, in kB
sudo grep '^Pss:' /proc/1234/smaps_rollup

Column	Meaning
USS	Memory unique to the process
PSS	Proportional share of shared pages + USS
RSS	Everything, double-counted if pages are shared

Swipe to see more

It is essential to diagnose a server that hosts many processes of the same application (Apache, PostgreSQL…).

Observe over time

vmstat, a snapshot every N seconds

vmstat prints one line per interval: here five readings two seconds apart.

vmstat 2 5

Essential columns for memory:

Column	Meaning
free	Unused memory
buff / cache	Buffers and cache
si / so	Swap in / out per second, live swap activity
r	Processes waiting to run
b	Processes blocked on I/O

Swipe to see more

The signal to watch

si/so columns that stay non-zero = the machine swaps continuously. Either the RAM is undersized, or a process leaks progressively. Never leave a production server in this state.

sar -r, history

sar (sysstat package) collects metrics continuously, once the collection is enabled (see the prerequisites), and lets you go back in time. The daily files live in /var/log/sysstat/ on Debian and Ubuntu, in /var/log/sa/ on RHEL:

# Real time: 3 readings 1 s apart
sar -r 1 3

# History of the day
sar -r

# History of a given day (/var/log/sa/ on RHEL)
sar -r -f /var/log/sysstat/sa$(date -d yesterday +%d)

Main columns: kbmemfree, kbmemused, %memused, kbbuffers, kbcached, kbswpfree, kbswpused, %swpused. Essential for an audit or a cold post-mortem.

Diagnose memory pressure

No single signal proves memory pressure: it is their combination that does. Cross these symptoms:

• free -h shows available < 10 % of total
• vmstat 2 shows non-zero si/so continuously
• dmesg -T | tail -50 contains Out of memory: Killed process
• The load average rises without the CPU being saturated, processes wait for memory

Chained diagnostic commands:

# who consumes?
ps aux --sort=-%mem | head -n 10

# since when?
sar -r | tail -30

# swap active?
vmstat 2 5

# has the OOM killer already struck?
sudo dmesg -T | grep -iE 'oom|killed'
journalctl -k --since "1 hour ago" | grep -iE 'oom|killed'

If the OOM killer struck, the typical line in dmesg is (format of recent kernels, truncated here):

Out of memory: Killed process 1234 (nginx) total-vm:512000kB, anon-rss:400000kB, file-rss:0kB, shmem-rss:0kB, UID:33 pgtables:1024kB oom_score_adj:0

You then have the PID, the process name, and its consumption at the time of the kill.

Act on memory

Diagnosis made, you can act. The levers split in two: adjust the policy (swappiness, adding swap) or free memory once (drop_caches).

Tune swappiness

vm.swappiness (0-100) sets how aggressively the kernel swaps. Typical values:

• 10, database server, latency critical
• 60, kernel default, workstation
• 30 or 10, values set by the tuned virtual-guest and throughput-performance profiles (read in their tuned.conf files): on a RHEL where tuned is active, check the actual value before concluding it is the default
• 100, maximum swap (rare)

Runtime change, lost at reboot, then check:

sudo sysctl vm.swappiness=10
cat /proc/sys/vm/swappiness

Persistence, through a dedicated file under /etc/sysctl.d/:

echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl --system

Add a swap file

Useful on a VPS without a swap partition. With a 1 Gi file, secured in mode 600:

1. Create the file and secure it

   sudo fallocate -l 1G /swapfile
   sudo chmod 600 /swapfile
   sudo mkswap /swapfile

2. Enable

   sudo swapon /swapfile
   swapon --show

3. Persist in /etc/fstab

   echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

4. Check after reboot

   sudo systemctl reboot
   # after reconnecting
   swapon --show
   free -h

Disable / re-enable temporarily

swapoff and swapon act on every swap area listed in /etc/fstab with -a:

# disable everything
sudo swapoff -a

# re-enable everything (per /etc/fstab)
sudo swapon -a

swapoff -a can freeze the machine if RAM is not enough to take back the swapped pages. Use with care.

Purge the cache (rarely useful)

sync first writes the dirty pages, then writing to drop_caches empties the cache:

sudo sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

Possible values: 1 (pagecache), 2 (dentry + inodes), 3 (everything). Useful for performance measurements or a cache rebuild test, not to solve memory pressure, Linux already frees the cache automatically.

Best practices

These rules sum up the traps measured above, in the order in which they cost the most: a misread figure triggers a false alarm, a forgotten sar collection leaves a post-mortem without data, a badly sized swap turns a peak into an incident.

• Read available, not free. 90 % of diagnostic mistakes come from there.
• Watch si/so of vmstat to detect chronic swapping.
• Keep sysstat enabled, sar saves the day for post-mortems.
• Do not purge the cache in production except for a targeted test. Linux handles it.
• Size the swap according to use: a few GiB are enough as a safety net on a well-sized server, and a database server with low swappiness only needs it to absorb a peak. There is no universal "swap = RAM" rule.
• Alert on MemAvailable and SwapUsed, not on MemFree.

Security and rollback

The only risky operation of the guide is swapoff -a while the swap is in use. Before disabling the swap:

# check that it is not in use
free -h
swapon --show

# if 'used' > 0, disable progressively or not at all

swappiness changes are reversible without risk. If a swap file is no longer wanted:

sudo swapoff /swapfile
sudo sed -i '/\/swapfile/d' /etc/fstab
sudo rm /swapfile

Quick troubleshooting

Read this table by symptom: the first two lines tell a dashboard false alarm from real memory pressure, which is always confirmed by a low available and an active swap at the same time.

Symptom	Likely cause	Action
A dashboard shows 90 %+ of RAM "used" but all is fine	The figure is (total - free), it includes reclaimable buff/cache	Read the used and available columns of free, not (total - free)
free shows a very high used	Non-reclaimable memory (used = total - available)	Check available and si/so: if low + swap active, real pressure
Low available + non-zero si/so	Real memory pressure	ps aux --sort=-%mem → identify and limit
The OOM killer killed a service	RAM+swap saturated	dmesg -T | grep -i oom for the victim, size RAM/swap or adjust the oom_score_adj of the critical service
Swap 100 % full but high available	Swap filled historically, not actively used	swapoff -a && swapon -a to empty it, only if RAM is comfortable
No swap on a cloud VPS	Image without swap, swapon --show empty	Create a swap file (see above)
htop / free inconsistent	Rare, ps cache vs direct /proc read	Cross-check with cat /proc/meminfo

Swipe to see more

Key points

• Read available, not free. It is the only right metric of available RAM.
• The cache always fills the RAM, it is healthy, not a problem.
• The critical fields of /proc/meminfo: MemAvailable, Buffers+Cached, SwapFree.
• ps aux --sort=-%mem and htop to find the hungry processes; smem -s pss (or Pss in smaps_rollup on RHEL) for the proportional share.
• With procps-ng 4.0.4, used is total - available: the two columns complement each other.
• vmstat 2: watch si/so. Non-zero continuously = chronic swap.
• sar -r keeps the memory history, essential for post-mortems, provided the collection is enabled (disabled by default on Debian and Ubuntu).
• vm.swappiness is adjustable at runtime (sysctl) and permanently (/etc/sysctl.d/).
• A swap file is useful on a VPS without a dedicated partition: fallocate + mkswap + swapon + an /etc/fstab entry.
• The OOM killer strikes when RAM+swap are saturated, the log is in dmesg and journalctl -k.
• Do not purge drop_caches in production except for a controlled test.

Knowledge check

Check that the essentials of this guide stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →