You are in the middle of an incident: a service went down, the server no longer answers, or the disk is full. This page is a lookup table. You find your symptom in it, the first command to type to qualify it, and the guide that handles the whole failure. Nobody reads a course during an outage: you look for your symptom, you want the command, and you want to know whether you fixed it.
This module is for the administrator who has to bring a service back in production, and for the one preparing the troubleshooting scenarios of the RHCSA and the LFCS, where diagnosis is timed.
Find your symptom
Each row of the table starts from what you observe, not from the technology at fault. Two rows carry the same message, "Permission denied", and that is deliberate: it is the only symptom in the list that is qualified in two steps. Unix permissions and ACLs are read first; if both are right and access is still refused, the cause is mandatory access control, SELinux on RHEL, AppArmor on Ubuntu. The second column is the command that qualifies the symptom, that is the one that turns a vague complaint into a verifiable fact. Until you have that answer, any fix is a guess.
| What you observe | The command that qualifies | The procedure |
|---|---|---|
The service refuses to start, systemctl reports it failed | systemctl status <service> | A service will not start |
The service is active (running) but no longer answers | curl -m 5 then ps -o stat | An active service that does not respond |
| You can no longer log in over SSH | Read the exact client message | Get SSH access back |
| The server is unreachable, or a port does not answer | ip addr then ss -tlnp | Diagnose a network problem |
| Writes fail, "No space left on device" | df -h then df -i | Disk space and inodes |
| The server is slow, for no obvious reason | uptime then vmstat 1 | Assess performance |
| The load average climbs and does not come down | uptime, top | Check system load |
| A process is killed without explanation | journalctl -k | grep -i oom | Monitor memory |
| The machine does not finish booting | Console, then journalctl -b -p err | A server stuck at boot |
| "Permission denied" on a file | ls -l then getfacl | Permissions and ACL |
| "Permission denied" although permissions and ACLs are right | ausearch -m avc -ts recent on RHEL, aa-status on Ubuntu | SELinux or AppArmor |
| A mount disappeared after a reboot | findmnt and /etc/fstab | Mount and make persistent |
| The file system refuses writes | touch: read the exact message | A read-only file system |
Method comes before commands
A troubleshooter in a hurry tries fixes at random and ends up breaking something else. The discipline that makes the difference fits in one sentence: observe, form a hypothesis, test it with a single change, verify, and only then move on to the next one. Without this method, you no longer know what fixed the failure, so you will be able neither to reproduce it nor to avoid it.
The guide to the Linux diagnostic method walks through this discipline step by step, from the symptom to the root cause: qualify, collect, isolate, test, validate, then document what fixed it.
The three sources that always talk
Whatever the symptom, three sources of truth answer first. The systemd journal tells what the system tried and why it gave up. The processes tell who is consuming and who is blocked. The sockets tell who really listens on the port you believe is open. A diagnosis that consulted none of the three is a diagnosis that guesses.
Each one has its guide. Read logs with journalctl teaches you to filter by unit, by priority and by boot. Manage processes shows how to identify a process, its state and its consumption, then stop it cleanly. Test connectivity uses ping, ss and curl to tell a closed port, a firewall and a silent application apart.
What you will learn
Troubleshooting is not a collection of commands, it is an observable capability: bringing a service back to a working state and proving that it is. By the end of this module, you know how to:
- Qualify an incident: turn "it no longer works" into a verifiable fact, with a command and its output.
- Read a systemd journal and spot the line that matters, not the fifty lines of noise around it.
- Isolate the faulty layer: network, service, configuration, storage or kernel, without touching the other four.
- Verify the repair, including after a reboot, because a repaired service that does not survive the reboot is not repaired.
Put it into practice
The method above is only worth something if it holds against a failure you did not choose. The lab draws one at random out of six and never says which: service stopped, wrong port, firewall closed, SELinux context, docroot permissions, full file system. You diagnose, you repair, and the repair must survive the reboot. Three checks reject heavy-handed repairs along the way: SELinux stays enforcing, firewalld stays active, and the docroot never becomes world-writable.
Key points
- The symptom is the way in, not the technology. You start from what the user observes and work back to the cause.
- Qualify before fixing: a command that turns the complaint into a fact, otherwise you repair an imaginary failure.
- One change at a time, otherwise you will never know which one fixed it.
- Three sources always answer: the systemd journal, the processes, the listening sockets.
- Repairing and proving are two distinct actions, and the proof includes getting through a reboot.