A server that does not finish booting

You rebooted a server after an intervention, and it never came back. SSH no longer answers, and on the console you find: You are in emergency mode. This page teaches you to identify what is blocking with one command, to repair from the console, and above all to resume booting without restarting. It also shows you why the two commands recommended everywhere, systemctl list-jobs and systemctl --failed, are of strictly no use in this situation.

The most frequent cause, by far: an /etc/fstab entry pointing to a missing device, without the nofail option. All the outputs below come from an AlmaLinux 10 server put into this state.

What you will learn

• Read what the console shows: three possible screens, three diagnoses
• Find the faulty mount or service with the only command that talks
• Repair from the emergency shell, including when the root is read-only
• Resume booting without restarting, with systemctl default
• Prevent: the option that would have avoided the incident

Understand what the console shows

Without SSH access, the console is your only window: the cloud provider's console, virsh console on a hypervisor, or a physical screen. What you read there steers the whole diagnosis.

What you see	What is happening	Where to go
A start job is running for... (45s / 1min 30s)	Boot is waiting for something: a device, a mount, a network. The counter goes up.	Wait for the delay to end: you will land in emergency mode, which gives you control
You are in emergency mode + Give root password	systemd has given up. It gives you a rescue shell.	It is the case handled here
A GRUB prompt, or nothing at all	The problem is before systemd: kernel, initramfs or boot loader	See Shut down, reboot, recover

Swipe to see more

The A start job is running message deserves an explanation, because it causes panic for nothing. The server has not crashed, it is waiting. systemd grants a 90-second delay by default to an expected device. Past this delay, it gives up and switches to emergency mode. Waiting is therefore the right action: after a minute and a half, you get a shell back.

Take control on the console

The emergency shell asks for the root password. Type it, and you get a prompt. You are now in a minimal system: the root is mounted, but almost nothing is started, and the network is not configured.

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit" to continue bootup.
Give root password for maintenance
(or press Control-D to continue):

No root password, no emergency shell

If the root account has no password (the default case on Ubuntu and on most cloud images), this prompt is a dead end: you cannot get in. You then have to go through GRUB and the initramfs (rd.break on RHEL, init=/bin/bash on Debian), described in Shut down, reboot, recover. Setting a root password on your servers is what makes this door usable the day you need it.

Find the culprit: only one command works

This is where most tutorials lead you astray. Here is what the two commands usually recommended give in a real emergency shell:

systemctl list-jobs
systemctl --failed --no-pager

No jobs running.
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

No job running. No failed unit. All is apparently well, while the server is on its knees. The explanation is simple: systemd had already dropped the pending jobs when it gave you the shell, and the mount unit did not go failed, it was cancelled. You are looking for a trace that no longer exists.

The only source that kept the memory of the incident is the journal of the current boot:

journalctl -b -p err --no-pager | tail

systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-deadbeef\x2d0000\x2d...
            .device - /dev/disk/by-uuid/deadbeef-0000-0000-0000-000000000000.

It is all there. systemd waited for a device identified by its UUID, never saw it appear, and ended up giving up. The UUID shown is the one you are about to find in /etc/fstab. Remember the -p err option: it filters on messages of "error" priority and saves you from reading hundreds of lines.

Then compare with the faulty file:

cat /etc/fstab

UUID=1f5fce98-2902-4ac3-b784-b4f10857f44e /        xfs   defaults  0 0
UUID=cea79fb2-b54a-4ab3-bd5e-25595813e238 /boot    xfs   defaults  0 0
UUID=deadbeef-0000-0000-0000-000000000000 /donnees ext4  defaults  0 2

The last line points to a disk that does not exist (or no longer does). A removed disk, a volume not attached after a migration, a UUID copied with a typo: the exact cause varies, the symptom is the same.

Repair

Check whether the root accepts writes

You read everywhere that you must start with mount -o remount,rw /. Check first, rather than applying a ritual:

findmnt -o TARGET,OPTIONS /

TARGET OPTIONS
/      rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

Here the root is already rw: the mount succeeded, it is another mount that failed. There is nothing to remount. If on the other hand you read ro in this output, then and only then:

mount -o remount,rw /

Neutralise the faulty line

Fix /etc/fstab with the available editor (vi is always there). Two options depending on your need: delete the line if the mount no longer has a reason to exist, or make it non-blocking with nofail if the device must come back later on.

vi /etc/fstab

Then reload the configuration. It is a step people forget, and the /etc/fstab file reminds you of it itself in its comments: systemd generates mount units from fstab, and it does not regenerate them on its own.

systemctl daemon-reload

Resume booting without restarting

No need to reboot: ask systemd to resume its startup where it had stopped.

systemctl default

Boot resumes, services start, and the server comes back online. Then check from SSH:

systemctl is-system-running
systemctl --failed --no-pager
grep deadbeef /etc/fstab

running
0 loaded units listed.

running means that all the boot units have reached their goal. If you get degraded, the system works but at least one unit has failed: systemctl --failed will tell you which one, usefully this time.

The previous boot stays readable

Once the server is back, the journal of the broken boot is not lost: journalctl -b -1 -p err shows the errors of the previous boot. It is precious for the incident report, and to check afterwards that you dealt with the cause and not just the symptom.

Prevent: the option that changes everything

The whole incident comes down to a missing option. An /etc/fstab entry without nofail is considered critical: if the device is missing, systemd refuses to finish booting. With nofail, the mount is attempted, and its failure does not prevent the boot:

UUID=... /donnees ext4 defaults,nofail,x-systemd.device-timeout=10 0 2

Two options, two complementary effects. nofail tells systemd that this mount is not vital: according to systemd.mount(5), boot continues without waiting for it. x-systemd.device-timeout=10 reduces the wait for the device from 90 seconds to 10: without it, even with nofail, the waiting job stays queued for a minute and a half when the disk is missing, and systemd only declares the boot finished once it expires.

The rule: every mount that is not essential to booting must carry nofail. Network shares, data disks, backup volumes fall into this case. The root and /boot do not: if they are missing, there is nothing to save.

And the reflex that will spare you the incident: after any change to /etc/fstab, test the mount without rebooting.

mount -a ; echo "exit=$?"
findmnt --verify

If mount -a fails, your next reboot will fail too. You have just learnt it while the machine is still reachable, which changes everything.

Troubleshooting

The table starts from what the console or the journal shows, and gives the check that names the cause. One row recalls the central trap of the page: in emergency mode, the usual commands no longer show anything.

Symptom	Likely cause	Verification	Fix
A start job is running that drags on	systemd is waiting for a device (90 s)	Wait: you will land in emergency mode	Fix the cause, add nofail
You are in emergency mode after a disk change	fstab entry without nofail	journalctl -b -p err: Timed out waiting for device	Delete the line or add nofail, then daemon-reload
list-jobs and --failed show nothing	The jobs were cancelled, not failed	journalctl -b -p err, the only reliable source	Do not rely on these two commands in emergency mode
The emergency shell refuses the root password	Root account without a password	Impossible to get in	Go through GRUB: rd.break (RHEL) or init=/bin/bash (Debian)
The system boots but shows degraded	A unit failed without blocking the boot	systemctl --failed (useful, this time)	Deal with the unit concerned
Boot only declares itself finished after 90 s, disk missing	nofail without x-systemd.device-timeout	systemctl list-jobs shows the device waiting job	Add x-systemd.device-timeout=10

Swipe to see more

Key points

• A start job is running is not a crash, it is a wait. After 90 seconds, systemd gives up and gives you a shell.
• In emergency mode, systemctl list-jobs and systemctl --failed show nothing: the jobs were cancelled, not failed. Do not waste time on them.
• journalctl -b -p err is the only command that names the culprit. Look for Timed out waiting for device.
• Check with findmnt -o OPTIONS / before remounting the root: it is often already rw, and the remount,rw ritual is then useless.
• systemctl daemon-reload after changing /etc/fstab: systemd generates its mount units from this file and does not regenerate them on its own.
• systemctl default resumes booting without restarting, from the emergency shell.
• nofail on every non-vital mount, together with x-systemd.device-timeout so that no job waits 90 seconds.
• mount -a and findmnt --verify after every fstab edit, while the machine is still reachable.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →