Linux diagnostic method

Picture this situation: you are responsible for a web application hosted on a Linux server. Since this morning, users have been complaining about slowness. Some requests take 30 seconds instead of 2.

You log in to the server and start looking:

• You run htop: the CPU looks normal, around 40%
• You check free -h: there is memory available
• You check the disk with df -h: 60% used, nothing alarming
• The Apache logs show nothing special

You are stuck. The problem is there, but you do not know where to look.

After two hours of searching, a more experienced colleague takes a look: "Did you check the I/O wait?". A glance at iostat shows that the disk is saturated with writes. A backup process, started by mistake in the middle of the day, is writing gigabytes. Problem found in 2 minutes by someone who knew where to look.

The difference between you and this colleague? A method.

What you will learn

• Follow a rigorous diagnostic approach, from the symptom to the root cause
• Recognise the four families of symptoms and their key commands
• Apply the 7-step method without skipping a step
• Avoid random changes and the classic anti-patterns
• Document an incident and its observations

What is symptom-based troubleshooting?

A structured approach, not chance

Symptom-based troubleshooting is a method that starts from what you observe (the symptom) and works back to what causes the problem (the root cause). It is the opposite of searching at random in the hope of stumbling on the solution.

A symptom is a visible manifestation of the problem. It is what you observe directly: the server no longer answers, the application is slow, a service refuses to start, the disk shows "full", or the network connection fails. The symptom is your starting point, your first clue.

A root cause is the real origin of the problem, what hides behind the symptom. For example, if the application is slow (symptom), it may be because a process consumes all the memory (root cause). If the disk is full (symptom), it may be because logs pile up without rotation (root cause). If the network connection fails (symptom), it may be because a firewall rule blocks the traffic (root cause).

The classic trap is to treat the symptom without looking for the cause. Restarting a service that crashes may make it work again for a while, but if you do not understand why it crashed, the problem will come back.

Why a method is essential

Without a method, troubleshooting becomes frustrating: unpredictable time, impossible to pass on to juniors, risk of making things worse.

With a method:

• You always know the next step
• Your approach is reproducible and can be documented
• You avoid dangerous actions
• You resolve faster and gain confidence

The four families of symptoms

Linux problems fall into four broad families: CPU, memory, disk and network. Identifying the right family immediately points to the right tools.

CPU symptoms: the processor is saturated

When the processor is overloaded, the system becomes slow or even stops responding. Commands take several seconds to run.

Going further

Read the process management guide to master top, htop, ps and process management. To understand why a process exists (its chain of ancestors), beyond what ps and lsof show, the witr tool traces back to the service or container responsible.

Common causes: runaway process (infinite loop), too many concurrent processes, DDoS attack.

Key commands:

htop                              # Interactive view
ps aux --sort=-%cpu | head -5     # Top 5 processes by CPU
uptime                            # Load average

Interpretation: load average > number of cores = overload. High %iowait = disk problem, not CPU.

Memory symptoms: RAM is saturated

Memory saturation shows up as a gradual degradation: an increasingly slow system, then processes killed by the OOM Killer.

Going further

Read the memory management guide to understand free, vmstat and how memory works on Linux.

Common causes: memory leak, too many simultaneous processes, undersized application.

Key commands:

free -h                           # Memory state
ps aux --sort=-%mem | head -5     # Top 5 processes by memory
journalctl -k | grep -i "oom"     # OOM Killer?

Interpretation: look at available (not free). High swap = not enough RAM. OOM Killer = note which process was killed.

Cache is not a problem

Linux uses free RAM as a disk cache to speed up file access. free shows it in its own column, buff/cache: an almost empty free column is therefore normal, the kernel gives that cache back as soon as a program asks for memory. The figure that matters is available.

Disk symptoms: storage is saturated

A full disk shows up as "No space left on device" errors. Writes fail, applications refuse to start.

Going further

Read the manage disk space and log management with logrotate guides to prevent these problems.

Common causes: logs piling up, temporary files, core dumps, backups never purged.

Key commands:

df -h                                    # Space per partition
du -sh /* 2>/dev/null | sort -rh | head  # Largest directories
lsof +L1                                 # Deleted but still open files

Interpretation: Use% at 100% = urgent action. Deleted files = restart the process to free the space.

Network symptoms: connectivity fails

Network problems are often the most stressful: SSH impossible, timeouts, "Connection refused".

Going further

Read the network diagnostics and understanding Linux networking guides to master network commands.

Common causes: service not started, firewall blocking, interface down, misconfigured DNS.

Diagnostic commands:

# Network interface state
ip addr
ip link

# Routing table
ip route

# Test connectivity
ping -c 4 8.8.8.8
ping -c 4 google.com  # also tests DNS

# Check that a port is open
ss -tlnp | grep :80
netstat -tlnp | grep :80   # older equivalent, missing from recent installs

# Firewall rules (nftables, the current filtering engine)
nft list ruleset
# or with firewalld
firewall-cmd --list-all

# DNS resolution
nslookup google.com
dig google.com

How to interpret:

• Interface DOWN: physical or configuration problem
• ping to IP OK but ping to name fails: DNS resolution problem
• Port not listed in ss -tlnp: the service is not listening
• reject/drop in the ruleset: the firewall blocks the traffic

The 7-step method

Here is the structured method to troubleshoot any Linux incident. Follow these steps in order, without skipping a step, even if you "think you know" where the problem is.

Overview

The diagram below reads from top to bottom, but above all remember the feedback loops: if step 6 (Test) refutes all your hypotheses, you do not move on to resolution, you go back to collecting data. Likewise, a problem you cannot reproduce at step 3 does not block you, it makes you prepare the instrumentation for the next occurrence. The first five steps are diagnosis and change nothing on the server; the test changes a single parameter at a time, and only the resolution applies the lasting fix.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Step 1: Observe, what is the exact symptom?

Do not rush to the keyboard. Start by properly understanding what is reported. This step is often botched, yet it shapes the whole approach.

Ask precise questions, to the user reporting the problem or to yourself. What exactly is happening? "It doesn't work" is not a usable symptom. Since when has the problem existed? Is it permanent or intermittent? If it is intermittent, how often, and have you spotted a pattern?

Who is affected: all users or only some? A single affected user may point to an account problem, a local network or a browser. All users affected point to the server.

What changed recently? A deployment, a system update, a configuration change? Finally, is there an error message? If so, write it down word for word: it is often the key to the diagnosis.

Example of a bad symptom:

"The server is slow"

Example of a good symptom:

"Since 2 pm, the site pages take 15 seconds to load instead of 2, for all users. Nothing deployed today."

Do nothing until you understand the symptom

Resist the temptation to "take a quick look" before you have properly understood the problem. Every minute spent clarifying the symptom saves ten in diagnosis.

Step 2: Collect, logs, metrics, context

Now that you know what to look for, gather the information.

System logs:

# Recent system logs
journalctl -xe --since "1 hour ago"

# Logs of a specific service
journalctl -u nginx --since "1 hour ago"

# Kernel logs (hardware problems, OOM, etc.)
dmesg -T | tail -50

# Application logs
tail -100 /var/log/nginx/error.log
tail -100 /var/log/apache2/error.log

System metrics:

# Quick overview
htop                     # CPU, memory, processes
df -h                    # Disk space
free -h                  # Memory
uptime                   # Load and uptime

# History if available
sar -u                   # CPU history
sar -r                   # Memory history

Context:

# Latest logins
last -10

# Latest commands (if .bash_history is accessible)
tail -20 ~/.bash_history

# Recent file changes
find /etc -mtime -1 -type f   # files modified in the last 24h

Step 3: Reproduce, is the problem reproducible?

A reproducible problem is ten times easier to solve than a ghost problem.

Being able to reproduce the problem gives you three major advantages. First, it confirms that the problem really exists and that you are not chasing a ghost. Then, it lets you test your hypotheses: if you think you have found the cause, you can check it by reproducing the problem under controlled conditions. Finally, after applying your fix, you can check that it works by trying to reproduce the problem and seeing that it is gone.

How to reproduce:

1. Identify the conditions

   When does the problem occur? Under what load? With what data?

2. Create a minimal test case

   What is the smallest action that triggers the problem?

3. Document the procedure

   Exact steps to reproduce, to share with colleagues.

If the problem cannot be reproduced:

Some problems are intermittent and refuse to show up when you look for them. In that case, get ready for the next occurrence. Set up collection scripts that automatically gather the useful data (logs, metrics, process state). Strengthen your monitoring to capture more detail. And above all, look for patterns: does the problem always happen at the same time? On the same day of the week? Under a certain load? These clues can reveal the cause.

Step 4: Isolate, which component is at fault?

A Linux system is made of layers stacked on top of each other. Identifying the faulty layer saves you from looking in the wrong place.

At the very bottom, the hardware: CPU, RAM, disks, network cards. Hardware problems often show up in dmesg as cryptic error messages. The smartctl tool checks disk health, and lshw gives a full view of the hardware.

Above it, the Linux kernel itself. Kernel problems appear in dmesg and journalctl -k. They are rare but serious: kernel panic, a bug in a driver, memory corruption.

Then comes the system layer: the base services managed by systemd, the init system. A systemctl status on the service concerned tells you whether it is running and whether it has had recent errors.

The network layer includes the interfaces, the firewall, the DNS. The ip, ss and nft tools let you diagnose this layer.

Finally, at the very top, your business application. That is often where the problem lies, and the application logs are your best source of information.

Isolation technique:

# Is the problem on THIS server?
# → Test the same service on another server

# Is the problem THIS service?
# → Test other services on the same server

# Is the problem the network?
# → Test locally (localhost) vs remotely

# Is the problem load-related?
# → Test with a single user

Step 5: Hypotheses, what are the possible causes?

Based on the symptom and the data collected, list the possible causes.

How to form good hypotheses:

Go from the most likely to the least likely. If a web server is slow, it is more likely that a process consumes too many resources than that there is a hardware problem on the motherboard.

Base your hypotheses on the data collected, not on your intuition. "I am sure it is the database" is not a hypothesis, it is a prejudice. A hypothesis looks like: "The database response time has risen by 500ms in the metrics, which could explain the slowness".

Consider what changed recently. Most incidents happen after a change: a deployment, an update, a configuration change. If something changed just before the problem appeared, it is a priority lead.

Example for "slow web server":

1. (Likely) A process consumes too many resources
2. (Likely) The database is slow
3. (Possible) The disk is saturated with I/O
4. (Possible) The network is congested
5. (Unlikely) Hardware problem

How to prioritise your hypotheses:

Three criteria to decide where to start. Probability first: start with what happens most often. Logs filling the disk are commonplace. A bug in the kernel is rare.

Next, ease of verification. Between two equally likely hypotheses, start with the one you can test in 30 seconds rather than the one that needs an hour of investigation.

Finally, impact. If a hypothesis, even an unlikely one, implies data loss or a security risk, check it quickly to reassure yourself or to act fast.

Step 6: Test, check each hypothesis

Test each hypothesis in order, one by one. It is tempting to try everything at once to go faster, but it is a trap.

The golden rule: one change at a time. If you change three things and the problem disappears, you do not know what solved it. Worse, you may have introduced two new problems while fixing the first.

Document each test as you go. Write down the exact command you ran, what result you expected, and what you actually observed. This record will be precious if you have to hand over to a colleague or come back to the incident afterwards.

Make sure you can roll back before each change. Do nothing irreversible without a backup. A configuration file must be copied before it is changed. A stopped service must be able to start again.

Structure of a test:

Hypothesis: The PHP-FPM process consumes too much memory
Test: ps aux --sort=-%mem | head -5
Expected result: PHP-FPM at the top of the list
Observed result: [what you see]
Conclusion: Hypothesis confirmed / refuted

If a hypothesis is confirmed:

Dig in that direction. Why does this process consume so much?

If all hypotheses are refuted:

Go back to step 2 (collect more data) or reformulate your hypotheses.

Step 7: Resolve, fix and document

Once the cause is identified, fix the problem.

Before fixing:

1. Assess the risk

   Could this action make the situation worse?

2. Prepare a rollback

   How do you go back if it does not work?

3. Warn the people concerned

   Tell the team what you are about to do.

4. Document what you are about to do

   Write the commands down before running them.

After fixing:

1. Check that the problem is solved

   Replay the initial scenario: has the symptom disappeared?

2. Watch the stability

   The problem may come back. Keep an eye on the metrics.

3. Document the resolution

   Cause, solution, time to resolve. This record will serve the next similar incident.

4. Share the lessons learned

   Inform the team. Write a postmortem if the incident was major.

Anti-patterns to avoid

The classic mistakes

These are the traps that even experienced administrators fall into:

• Restarting without understanding: the problem will come back and you will have lost the chance to collect data. Diagnose first.
• Running commands at random: wasted time and risk of making things worse. Follow the method.
• Not reading the logs: the answer is often in there. journalctl -xe should be your first reflex.
• Changing production without a backup: always copy a file before changing it. Always.
• Working alone for too long: after 30 minutes without progress, ask for help. A fresh pair of eyes sees what you miss.
• Not documenting: write down what you do during the incident. Your future self will thank you.
• Assuming without checking: "I am sure that's it" is not a diagnosis. Test your hypotheses.
• Several changes at once: impossible to know what worked. One change, one test.
• Ignoring warnings: they become errors, and errors become outages.
• Skipping the postmortem: you will repeat the same mistakes. Analyse and document.

Anti-pattern vs good practice

This diagram contrasts two paths starting from the same incident. What separates them is not technical skill but the moment the administrator touches the server: on the haphazard branch, the first action changes the state before any diagnosis, and destroys along the way the traces that would have made it possible to understand. On the methodical branch, the first action is a read. A single rule sums up both: as long as you have no cause, you observe, you do not fix.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Troubleshooting checklist

Two checklists, two uses. The first is run during the incident, without leaving the terminal; the second frames an incident that involves several people and will have to be justified afterwards. Choose according to the question "will I have to account for this incident?".

Quick checklist (simple incidents)

Seven points to go through from top to bottom on an isolated incident, single service, no major customer impact. The point most often skipped is the last one, documentation: nobody does it once the service is back up. Yet it is what turns a successful repair into a reusable procedure.

• Symptom clearly identified
• Logs read (journalctl, dmesg)
• Resources checked (CPU, memory, disk)
• Hypothesis formed
• Solution tested
• Resolution verified
• Documentation updated

Full checklist (major incidents)

On a major incident, half of the points below are not about technique but about communication and traceability. Two of them must be prepared before touching the system, not after: the backup and the rollback. And the timeline is built as you go: rebuilding it from memory two days after gives a wrong postmortem.

• Symptom documented with timestamp
• Affected people identified
• Communication sent ("incident in progress")
• Logs collected and saved
• Historical metrics analysed
• Timeline established (when it started, what changed)
• Hypotheses listed and prioritised
• Each test documented (command, result)
• Backup taken before changes
• Rollback prepared
• Solution applied
• Post-resolution check
• End-of-incident communication
• Postmortem scheduled
• Preventive measures identified

Key points

1. No panic, method first

   Breathe. Follow the steps in order. Haste makes problems worse.

2. The symptom is not the cause

   "Slow server" is a symptom. The cause is elsewhere (runaway process, saturated disk, etc.). Look for the root cause.

3. Logs are your friends

   A large share of incidents leave an explicit trace in the logs. journalctl and dmesg before any other command.

4. One change at a time

   If you change three things and it works, you do not know what solved the problem.

5. Document everything

   Your 3 am self will thank you for writing down the solution last time.

6. Asking for help is not a failure

   After 30 minutes without progress, escalate. Two pairs of eyes are better than one.

7. Prevention beats cure

   The best troubleshooting is the one you do not have to do. Monitoring, alerts, documentation.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

External resources

• USE Method, Brendan Gregg, the Utilization-Saturation-Errors method for performance diagnosis
• Linux Performance, Brendan Gregg, Linux performance tools and methods
• Site Reliability Engineering, Google, chapter on incident management
• Red Hat System Administration Guide, official RHEL documentation

×

See in the glossary →