Get SSH access back to a server

You have just changed the SSH configuration of a remote server, and you can no longer connect to it. The most stressful failure in Linux administration, because the tool you use to repair is precisely the one that is broken. This page teaches you to qualify the refusal first, then to handle the three causes that produce three different symptoms, and finally the rule that would have spared you the incident.

All the commands and outputs on this page come from an AlmaLinux 10 server with SELinux in Enforcing, the default configuration of the RHEL world.

What you will learn

• Qualify the refusal: refused, timeout and Permission denied point to three distinct culprits
• Validate sshd_config with sshd -t before any restart
• Diagnose the non-standard port trap blocked by SELinux, one of the classic RHCSA traps
• Fix a key rejection caused by the permissions of authorized_keys
• Keep a way out so you never lock yourself out

Qualify the refusal before touching the server

The error message of your SSH client is, on its own, the diagnosis. Three refusals, three causes, three procedures: mixing them up means looking in the wrong place.

What the client says	Meaning	Likely culprit
Connection refused	The server answered, but nothing listens on this port	sshd is stopped or listens elsewhere
Nothing, then Connection timed out	No answer: packets are silently dropped	Firewall, security group, or server powered off
Permission denied (publickey)	The connection works, it is authentication that fails	Key, permissions of authorized_keys, or account

Swipe to see more

A refusal is not a timeout. A refusal proves that the machine is alive and reachable: the network is fine, the problem is on the service. A timeout, on the other hand, says nothing about the service: the packets do not even reach it. And a Permission denied is almost good news: everything works, except the proof of identity.

Never close your current session

As long as you have an open SSH session, you have a door. It survives a systemctl restart sshd (the process serving you is already forked). Open a second connection to test your changes, and only close the first one once the second works. It is the rule that separates a ten-second incident from a trip to the server room.

Cause 1: a syntax error in sshd_config

It is the most common mistake: a misspelled directive, and the service refuses to restart. The important point is that this failure is entirely avoidable, with a single command.

The command that saves you

sshd -t analyses the sshd configuration without touching the running service. It returns 0 if all is well, and a non-zero code otherwise:

sshd -t ; echo "exit=$?"

exit=255

A non-zero exit code, and you do not restart. You fix first. If you restart anyway, here is what happens:

systemctl restart sshd ; echo "exit=$?"
systemctl is-active sshd
ss -tlnp | grep -E ':22\b' || echo "AUCUN processus n'ecoute sur le port 22"

Job for sshd.service failed because the control process exited with error code.
exit=1
inactive
AUCUN processus n'ecoute sur le port 22

The service is stopped, nothing listens on port 22 any more, and any new connection will get a Connection refused. Your current session, for its part, survives: it is your only way out.

Repair

Restore the backup of your configuration (you made one before changing it), validate, then restart:

cp -a /root/sshd_config.orig /etc/ssh/sshd_config
sshd -t && echo "sshd_config valide"
systemctl restart sshd
ss -tlnp | grep -E ':22\b'

sshd_config valide
LISTEN 0      128          0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1865,fd=7))

Cause 2: the non-standard port blocked by SELinux

Changing the SSH listening port is a common request. On Debian or Ubuntu, editing sshd_config is enough. On RHEL, AlmaLinux or Rocky, it is not enough, and the error you get is particularly misleading.

The syntax is valid, and yet the service refuses to start

sed -ri 's/^#?Port .*/Port 2222/' /etc/ssh/sshd_config
sshd -t ; echo "exit=$?"
systemctl restart sshd ; echo "exit=$?"

exit=0
exit=1

Look closely: sshd -t returns 0. The configuration is perfectly valid. And yet the restart fails. It is exactly the kind of contradiction that costs half an hour, because the validation command you have just learnt tells you here that all is well.

The journal gives the cause, but it is misleading

journalctl -u sshd --since '1 min ago' --no-pager | tail -3

sshd[1816]: error: Bind to port 2222 on 0.0.0.0 failed: Permission denied.
sshd[1816]: error: Bind to port 2222 on :: failed: Permission denied.

"Permission denied" while you are root. That is the signal. Root never has a Unix permission problem opening a port. When root is refused something for no apparent reason, the number one suspect is mandatory access control, that is SELinux.

The AVC record names the culprit

SELinux logs each refusal as an AVC (Access Vector Cache) record:

ausearch -m avc -ts recent | tail -2

type=AVC msg=audit(1784022633.371:656): avc:  denied  { name_bind } for  pid=1816
comm="sshd" src=2222 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0

This line reads like a sentence. The sshd daemon (in the sshd_t context) attempted a name_bind on port 2222, whose label is unreserved_port_t. SELinux only allows sshd_t to bind to ports labelled ssh_port_t. Port 2222 is not one of them, so the bind is refused.

Check which ports the policy really knows:

semanage port -l | grep -E '^ssh_port_t'

ssh_port_t                     tcp      22

semanage is not installed by default

On an AlmaLinux or RHEL cloud image, SELinux is in Enforcing but the tools to drive it are not there. semanage and audit2why come with the policycoreutils-python-utils package. Install it before you need it: a dnf install in the middle of a failure, on a server you have just lost access to, is a bad moment.

Fix

You label the port for the SELinux policy, then restart:

semanage port -a -t ssh_port_t -p tcp 2222
semanage port -l | grep -E '^ssh_port_t'
systemctl restart sshd ; echo "exit=$?"
ss -tlnp | grep -E ':2222\b'

ssh_port_t                     tcp      2222, 22
exit=0
LISTEN 0      128          0.0.0.0:2222      0.0.0.0:*    users:(("sshd",pid=1847,fd=7))

Port 2222 is now known to the policy, sshd binds to it, and the service starts. Two steps remain before closing your witness session: open the port in the firewall (firewall-cmd --permanent --add-port=2222/tcp, then firewall-cmd --reload to apply it) and check that a new connection does get through on the new port. An SSH port changed without a matching firewall rule produces a timeout, and you are back in the first column of the qualification table.

Cause 3: the permissions of authorized_keys

Your key is the right one, it is indeed in authorized_keys, and yet the server refuses it. The cause is almost always StrictModes, enabled by default: sshd refuses to use a key file that other users could modify. The logic is sound: an attacker who can write to your authorized_keys can add their own key to it.

The symptom on the client side

ssh -i /root/.ssh/lab_probe root@127.0.0.1 'echo connexion OK' ; echo "exit=$?"

root@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
exit=255

Nothing in this message talks about file permissions. The SSH client does not know why the server refused its key: the server does not tell it, and that is deliberate (a talkative server would help an attacker).

The cause is on the server side, in the journal

journalctl -u sshd --since '1 min ago' | grep -i 'authentication refused'
stat -c '%a %U:%G %n' /root/.ssh/authorized_keys

sshd-session[1910]: Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
777 root:root /root/.ssh/authorized_keys

"bad ownership or modes", the server journal is explicit where the client was silent. The file is 777: anyone can rewrite it, so sshd ignores it. Remember this reflex: when a key is refused, the real message is on the server side, in journalctl -u sshd.

Fix

chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys
chown -R root:root /root/.ssh
restorecon -Rv /root/.ssh

The permissions to aim for are 600 for authorized_keys and 700 for the .ssh directory: it is the hygiene target, with no possible ambiguity.

But do not diagnose with this rule, because it is too strict and would make you look for a problem where there is none. Measured through real connections on AlmaLinux 10 with OpenSSH 9.9p1:

Situation	Connection
authorized_keys in 600, 644 or 640	accepted
authorized_keys in 660, 664, 606, 666	refused
~/.ssh in 700	accepted
/home/<user> in 770	refused, although ~/.ssh is perfect

Swipe to see more

What triggers a refusal is the write permission for the group or for others, and the owner, never mere readability. An authorized_keys in 644 therefore works perfectly: if your connection fails, the cause is elsewhere.

Second lesson, less well known: the check goes up to the home directory. A /home/<user> in 770 is enough to make the connection fail, and it is a cause nobody ever looks for, busy as we are inspecting ~/.ssh over and over. The restorecon also restores the correct SELinux labels, often lost when the file was moved with mv (or copied with cp -a) from another location: these two commands keep the original context, whereas a plain cp applies the one of the target directory.

Troubleshooting

The table starts from the client message, then gives the check on the server side that confirms the cause. The last two rows cover rarer cases: OpenSSH's throttling after several failures, and the server that no longer boots at all.

Symptom	Likely cause	Verification	Fix
Connection refused	sshd stopped (invalid config)	systemctl is-active sshd, sshd -t	Fix the config, systemctl restart sshd
Connection refused after a port change	The service listens elsewhere	ss -tlnp | grep sshd	Connect with ssh -p <port>
Connection timed out	Firewall or security group	firewall-cmd --list-all	Open the port, also check the provider's firewall
The service refuses to start on a non-standard port, sshd -t says 0	SELinux: port not labelled	ausearch -m avc -ts recent	semanage port -a -t ssh_port_t -p tcp <port>
Permission denied (publickey) with the right key	StrictModes: write permission for the group or others	journalctl -u sshd, stat on authorized_keys and on /home/<user>	chmod 600 on the file, 700 on .ssh and on the home directory
Permission denied (publickey), journal silent about modes	The file belongs to root: sshd reads it after dropping its privileges	Look for Could not open user … authorized keys in the journal	chown <user>:<user> on .ssh and its content
The symptom changes during diagnosis, kex_exchange_identification: read: Connection reset by peer	PerSourcePenalties (OpenSSH 9.8+) has throttled you after several failures	sudo sshd -T | grep persourcepenalties	Wait: the default setting is authfail:5 min:15, it wears off in less than a minute
Everything looks right but nothing gets through	The server no longer boots	Provider console or virsh console	See Shut down, reboot, recover

Swipe to see more

Put it into practice

The sshd -t before the restart reflex only sinks in by living it on a machine where the mistake costs nothing. The lab confronts you with a broken sshd_config: you qualify the refusal, repair the configuration and validate the service's return, without ever closing your witness session.

Guided lab Practice

Learn how to do it

Repair a broken sshd config before it locks you out

$ dsoxlab run l3-ssh-access-recovery

Environment: VM·Duration: ~25 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start l3-ssh-access-recovery checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/linux-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run l3-ssh-access-recovery
dsoxlab check l3-ssh-access-recovery

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

• The client message is the diagnosis: refused (nothing listens), timeout (firewall), Permission denied (authentication). Three causes, three procedures.
• sshd -t before every restart. This command alone prevents most lockouts.
• Always keep a session open and test with a second connection. An established session survives the restart of sshd.
• On RHEL, changing the SSH port is not enough: SELinux only allows sshd to bind to ssh_port_t ports. The symptom is a Permission denied while you are root, and sshd -t does not see it.
• semanage is not installed by default on RHEL cloud images, whereas SELinux is indeed in Enforcing.
• A refused key is diagnosed on the server side: the client never says why. journalctl -u sshd reveals the "bad ownership or modes".
• authorized_keys in 600, .ssh in 700. A write permission for the group or others, and StrictModes ignores the file.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →