An active service that does not respond

systemctl status shows active (running), and yet the service no longer answers. It is the worst case, because the tool you rely on lies by omission: systemd knows the process exists, it does not know whether it does its job. This page gives you the command that settles it in five seconds, then the two causes hiding behind this symptom, with their respective signatures.

This page complements A service will not start, which handles the case where the unit is failed. Here, the unit is green: that is the whole problem, and the diagnosis must start elsewhere.

What you will learn

• Qualify the symptom with curl and read its exit code, which on its own names the family of the failure
• Recognise a frozen process with ps, /proc and strace
• Check the real listening address of a service, and understand why nginx -t will never see this error
• Repair in both cases, and prove that the service answers again

The curl exit code settles it first

Before touching the server, ask the one question that steers everything else: is the connection refused or hanging? Both produce "it doesn't work" on the user side, but they have nothing in common. Tested locally, a client that waits forever has managed to establish the connection, so something is listening. A client refused outright found nobody on the port.

curl -s -o /dev/null -w 'http %{http_code} en %{time_total}s\n' -m 5 http://127.0.0.1/
echo "code de sortie : $?"

Two possible answers, two opposite diagnoses:

Observed output	Code	What it means	Lead
http 000 en 5.002338s	28	The connection is established, but nothing answers. The client waited until the end of -m 5.	The process is frozen or blocked
immediate http 000	7	Connection refused: nobody listens on this port or address.	Wrong listening address, or service stopped

Swipe to see more

The response time is the signal. Five full seconds is a timeout, so a process that no longer handles requests. An instant failure is the kernel sending back an RST because no socket listens where you are knocking.

From another machine, a code 28 has a second explanation: a firewall that drops packets (a DROP rule) also makes the client wait until the end. curl's %{time_connect} variable tells the two apart, by saying whether the TCP connection was established:

curl -s -o /dev/null -w 'connect %{time_connect}s, total %{time_total}s\n' -m 5 http://10.0.2.15/

Measured with curl 8.12 on AlmaLinux 10: a frozen process gives connect 0.000076s (connection established, nothing answers), dropped packets give connect 0.000000s after the full delay (the connection never succeeded).

Do not trust systemd's verdict

systemctl is-active answers active in both cases above. systemd watches the process life cycle, not its ability to serve. A process that is stopped, blocked writing to a dead disk or waiting for an unreachable database remains, for systemd, perfectly "active".

Case 1: the process is frozen

A frozen process uses no CPU, logs nothing, and keeps its port open. It is the most confusing case, because all your usual checks turn green. It happens when the process receives a SIGSTOP, when it waits for an I/O that never comes back (dead NFS mount, failing disk), or when it is stuck on a lock.

What systemd and ss tell you (and it is misleading)

systemctl is-active nginx
systemctl status nginx --no-pager | head -6

active
     Active: active (running) since Tue 2026-07-14 09:50:20 UTC; 3s ago

The service is declared alive. The port, too, is still listening:

ss -tlnp | grep -E ':80\b'

LISTEN 1      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=1667,fd=6),...)

One detail still gives the failure away. The first column, Recv-Q, is 1 instead of 0. On a listening socket, Recv-Q counts the connections accepted by the kernel but never picked up by the application. The kernel does its work, the application no longer comes to fetch its clients. A queue growing on a LISTEN socket is the signature of a process that no longer handles requests.

The proof is in the process state

A process state is a letter, and that letter tells you everything. Get the main PID from systemd, then query the kernel:

MAIN=$(systemctl show -p MainPID --value nginx)
ps -o pid,stat,wchan:20,cmd -p "$MAIN"
grep -E '^(Name|State):' "/proc/$MAIN/status"

   1667 Ts   do_signal_stop       nginx: master process /usr/sbin/nginx
Name:  nginx
State:  T (stopped)

T means stopped. The process is suspended, it will not execute a single instruction until it is woken up. The wchan column even states which kernel function it is sleeping in: do_signal_stop. The other states to know: R (running), S (sleeping, normal), and above all D (uninterruptible sleep), which signals a blocked I/O and which, for its part, cannot be fixed with a signal.

Check the whole service, not only the main process. A server like nginx delegates the work to workers:

ps -o pid,stat,cmd -C nginx

   1667 Ts   nginx: master process /usr/sbin/nginx
   1668 T    nginx: worker process
   1669 T    nginx: worker process

All three processes are frozen. If only the master were, the workers would keep answering and you would see no failure.

Confirm with strace

strace attaches to the process and shows the system calls it makes to the kernel. On a healthy process, the output scrolls. On a frozen process, it is empty, and the tool tells you why:

timeout 5 strace -p "$MAIN"

strace: Process 1667 attached
--- stopped by SIGSTOP ---

The diagnosis is written in black and white. Note that strace and lsof are not installed on a minimal cloud image: a dnf install -y strace lsof (or apt install) is better done before the incident.

The journal, meanwhile, stays hopelessly silent, and that is logical: a frozen process cannot write its own logs. No trace does not mean no problem.

Repair

If the cause is a stop signal, you resume the whole process group of the service. systemctl kill acts on the entire cgroup, so on the master and all its workers, which a kill on the main PID alone would not do:

systemctl kill --signal=SIGCONT nginx

Then you prove the repair, which is not the same thing as noticing the absence of errors:

ps -o pid,stat,cmd -C nginx
curl -s -o /dev/null -w 'http %{http_code} en %{time_total}s\n' -m 5 http://127.0.0.1/

   1667 Ss   nginx: master process /usr/sbin/nginx
   1668 S    nginx: worker process
http 200 en 0.000454s

The states are back to S, and the answer comes back in less than a millisecond. If the state had been D and not T, SIGCONT would have changed nothing: you would have had to deal with the blocked I/O, that is the dead network mount or the failing disk.

Case 2: the service listens on the wrong address

This case is commonplace, and yet it costs hours because it works perfectly locally. You test from the server, all is well. Clients, for their part, get an immediate refusal. The cause is a listen directive limited to the loopback instead of all interfaces.

The trap: the configuration is valid

grep -nE 'listen' /etc/nginx/nginx.conf
nginx -t

38:        listen 127.0.0.1:80;
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

nginx -t validates the syntax, not the relevance. The file is correct, the service starts without a single error, and the journal is empty. No validation tool will ever tell you that listening on 127.0.0.1 is an operational mistake: it is a perfectly legitimate configuration, just not the one you wanted.

The symptom depends on where you test from

curl -s -o /dev/null -w 'sur 127.0.0.1: http %{http_code}\n' -m 5 http://127.0.0.1/
curl -s -o /dev/null -w 'sur 10.0.2.15: http %{http_code}\n' -m 5 http://10.0.2.15/ ; echo "exit=$?"

sur 127.0.0.1: http 200
sur 10.0.2.15: http 000
exit=7

From the server: 200. From its network address: immediate refusal, exit code 7. It is the very definition of the "works on my machine" symptom. Always test from the address your clients come through, never from localhost.

ss gives the answer in one line

ss -tlnp | grep -E ':80\b'

LISTEN 0      511        127.0.0.1:80        0.0.0.0:*    users:(("nginx",pid=1737,fd=6))

The listening address is 127.0.0.1:80 and not 0.0.0.0:80. 0.0.0.0 means all interfaces, 127.0.0.1 means the loopback only. Fix the directive, reload, and check that listening is back on all interfaces before declaring the incident closed.

Troubleshooting

The table crosses the signatures seen above: the curl exit code, the process state and the Recv-Q column of ss. Each row gives the verification that confirms the cause before the fix, never the other way round.

Symptom	Likely cause	Verification	Fix
curl exits with 28 after the full delay	Process frozen (T) or blocked on I/O (D)	ps -o stat and /proc/<pid>/status	systemctl kill --signal=SIGCONT if T; deal with the mount or the disk if D
curl exits with 28 from the network, time_connect at 0	A firewall drops the packets	curl's %{time_connect}, then the firewall rules	Open the port in the firewall
curl exits with 7 immediately	Nothing listens on this address	ss -tlnp: compare the listening address	Fix the listen directive, reload
Recv-Q grows on a LISTEN socket	The application no longer accepts its connections	ss -tlnp	See the frozen process case
The service answers locally, not from the network	Listening on 127.0.0.1	ss -tlnp	Listen on 0.0.0.0 or the intended IP
The service answers, but very slowly	Saturation, not blocking	uptime, vmstat 1	See Assess performance
The journal is empty while the service is broken	Frozen process: it can no longer log	ps -o stat	Do not conclude "no log, no problem"

Swipe to see more

Key points

• The curl exit code steers the diagnosis: 28 (timeout) means you are waiting for a process that no longer answers, or for dropped packets (time_connect at 0), 7 (refused) means nobody listens where you are knocking.
• active (running) proves nothing. systemd watches whether the process exists, not its ability to serve.
• The process state is a letter: T (stopped) is fixed with SIGCONT, D (blocked I/O) is not fixed by a signal, you must deal with the cause.
• A non-zero Recv-Q on a listening socket is the signature of an application that no longer accepts its connections.
• A frozen process does not log. An empty journal during a failure is a piece of information, not a relief.
• nginx -t validates the syntax, not the listening address. Always check with ss -tlnp that the service listens where clients arrive.
• Install strace and lsof before the incident: they are missing from minimal cloud images.

Knowledge check

Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →