For an HAProxy entry point to survive a failure, you need two machines that serve the same configuration and a certificate of the same trust, an address that moves from one to the other, and above all a failover that watches the service and not the machine. This lesson is for someone who already runs an HAProxy 3.4 and wants to make it tolerant to the loss of a node (each of the two machines). It explains the failure domains (what goes down at the same time), the trap of the vrrp_script (the test that removes the address from a node whose HAProxy is dead) and the choice of its weight, the split-brain (two machines that each believe they alone hold the address), what must stay identical between the two nodes, then how to measure a recovery without touching the client.
What you will learn
- Tell apart the loss of a process from that of a host, and say what each test proves.
- Explain why a failover must watch the service and not the machine, tell apart the two paths of the weight (0 or negative) and compute a negative weight that really triggers the failover.
- Choose between a floating address, an upstream entry point and DNS according to what the network allows.
- Write the keepalived configuration of one node, then adapt the second one's without breaking the election.
- Recognize a split-brain as soon as the setup is done, and diagnose a double holder with
journalctlandip. - List what must stay identical between the two nodes.
- Measure a recovery with a probe that changes nothing on the client side.
Prerequisites
You have an HAProxy 3.4.7 that serves a service (the lab version, installed from trixie-backports-3.4 on Debian 13), as in installing HAProxy, and you know how to read the state of a backend server and its health checks, as in server health with HAProxy. You are comfortable with systemctl, ip addr and curl. The notion of reverse proxy is laid out in the reverse proxy guide. For an overview of the other high availability building blocks (databases, storage, clusters), see the high availability section (in French).
Is a single HAProxy enough to guarantee availability?
No: a single HAProxy is a single point of failure, and two instances only protect against the failures they do not share. A failure domain is the set of everything that goes down at the same time when one element fails. The haproxy process is one: it can crash or be killed without the machine moving. The host is another: a VM that reboots, a full disk, a cut power supply take down every process it hosts. Higher still, the upstream network (the switch, the gateway, the public address) is a third.
Two HAProxy instances on the same machine cover the first domain and leave the second intact: the host failure takes them down together. To prove that an entry point survives the loss of a host, you therefore need two distinct hosts, which is why the lab of this lesson starts three VMs: edge.lab and edge2.lab for the two entry points, poste.lab for the client that queries from the outside. A single machine would only prove the loss of a process.
When can you use a floating address, and what do you do otherwise?
A floating address (VIP) driven by VRRP is suitable when the two machines share a layer 2 network (they reach each other directly, without a router between them) that accepts an address changing holder; if the network forbids it, you choose an upstream layer 4 entry point or DNS, with their limits. The principle: the service address is wired to no machine in particular, the elected node adds it to its interface. VRRP (Virtual Router Redundancy Protocol) designates this holder: each node has a priority, a configured number, and the one with the highest priority holds the address and regularly announces its presence; if the holder goes silent, the other takes the address. keepalived is the daemon that implements VRRP on Linux. Clients change nothing: they always reach the same address.
Not every network accepts this. A network that does not route multicast (sending a packet to a group of subscribed machines, which VRRP relies on by default, group 224.0.0.18), or that forbids an address changing holder, makes the solution unusable; check on your network before prescribing it. This lesson does not list the clouds concerned. keepalived can announce in unicast, from one node to the address of the other, which works around the multicast block but not a ban on moving the address. The two alternatives each have a limit: a layer 4 load balancer upstream (which distributes the TCP connections in front of the two HAProxy) becomes itself the single point to protect, and DNS with several records does not remove a dead address until the clients' cache has expired.
Why must keepalived watch HAProxy and not only the machine?
Without monitoring of the service, keepalived only controls the machine: an HAProxy dead on a living machine triggers no failover, and the address stays on a node that no longer serves anything. This is the central trap of proxy high availability. keepalived announces that its holder is alive as long as its own process answers, whether HAProxy is running or not. The service is down, the high availability "is doing fine", and nobody is alerted.
The remedy is a vrrp_script: a test that keepalived runs at regular intervals, attached to the VRRP instance by track_script. Its weight decides what happens when this test fails, and two paths exist. Left at its default value, 0, the failure puts the instance in the FAULT state (the "faulted" state of a VRRP instance, neither master nor backup), with no priority computation at all. The keepalived.conf(5) manual says so plainly: "The default weight equals 0, which means that any VRRP instance monitoring the script will transition to the fault state".
With a negative weight, keepalived lowers the node's priority by the absolute value of the weight when the test fails fall times in a row (fall is the number of consecutive failures tolerated), which the same manual words as "subtracted from the initial priority in case of fall failures". This lesson details the negative weight, but weight 0 is just as legitimate a choice. Weight 0 is simpler (no priority to compute) and faster: the node goes to FAULT, stops announcing (it then sends a priority of 0, which the log notes as "sent 0 priority" only under -D, see below) and the peer becomes master in about 1 second. The negative weight (-60 on a node of priority 150, that is 90 against a peer at 100) switches over in about 3 seconds, but keeps the node as a candidate during its illness: it stays in the election with a lowered priority instead of leaving it. In both cases, the healed node comes back. These durations were measured on keepalived 2.3.3 in a container (two nodes in unicast, priorities 150 and 100, interval 1, fall 2, rise 2), on 2026-10-26: the manual does not state them, replay them on the lab.
After healing, the node takes the address back about 8 seconds later, by preemption (the node with the highest priority takes the address back as soon as it is a candidate again): this produces a brief second outage, which the nopreempt keyword avoids for whoever does not want it.
The lowering only has an effect if it brings the priority below that of the peer, otherwise the sick node keeps the address.
The table shows it on an example: the holder node has priority 150, its peer 100 (a gap of 50), and the holder's test fails.
| Weight | Priority of the sick node | Priority of the peer | Result |
|---|---|---|---|
| -20 | 130 | 100 | No failover: 130 stays above 100 |
| -50 | 100 | 100 | Tie: the result depends on a tie-break that is hard to read |
| -60 | 90 | 100 | Failover: the other node takes the address |
| 0 (default) | not applicable | 100 | Instance in FAULT state after fall failures: priority no longer comes into play |
If you keep the negative weight, therefore choose a weight whose absolute value exceeds the gap between the two priorities, with some margin: at equal priorities, VRRP breaks the tie by the highest primary IP address (RFC 5798, section 6.4.3), which makes the result depend on the addresses rather than on the failure. The manual accepts a weight from -253 to 253. The priority computation comes from the manual and the arithmetic above; the failover durations come from the container measurement quoted earlier, not from the lab.
Look at the diagram from top to bottom: the failure is the same, only the decision to watch the service changes the outcome. The test itself can be minimal (a pgrep -x haproxy checks that a process of that exact name exists) or more faithful (a real request on the local port, which proves that HAProxy answers). The more the test resembles what a client does, the smaller the gap between "the failover thinks all is well" and "the service is usable". A blocked, saturated or no longer listening HAProxy would pass the process test, where a real request would fail. Whatever command you choose, it must exist on each node: see below.
What does the keepalived configuration of a node look like?
A node's configuration fits in /etc/keepalived/keepalived.conf: a vrrp_script block that tests HAProxy, a vrrp_instance block that describes the election, and a track_script that links the two. The following example uses other names and another addressing plan than your network: two proxies proxy-a (172.16.40.11) and proxy-b (172.16.40.12) share the service address 172.16.40.100, on the ens18 interface. Here is proxy-a, the preferred node, in unicast.
vrrp_script check_haproxy { script "/usr/bin/pgrep -x haproxy" interval 1 fall 2 rise 2 weight -60}
vrrp_instance VI_WEB { state BACKUP interface ens18 virtual_router_id 77 priority 150 advert_int 1 unicast_src_ip 172.16.40.11 unicast_peer { 172.16.40.12 } virtual_ipaddress { 172.16.40.100/24 } track_script { check_haproxy }}The vrrp_script block follows keepalived.conf(5). script is the command run, a non-zero exit code being a failure: if it is wrong or missing, the test fails at every run and the node withdraws by itself. interval sets the seconds between two tries, fall the number of consecutive failures before declaring the test failed, rise the number of successes before declaring it restored. weight is -60 here: with 150 against 100, the priority drops to 90, below the peer's, as the table above shows.
The example uses pgrep -x haproxy (the -x requires the exact name) because pgrep comes from the procps package, of priority "important" on Debian, and therefore present on a minimal installation. It returns a non-zero code if no process matches, which makes it a test. The classic alternative, killall -0 haproxy, comes from the psmisc package, of priority "optional": it requires installing it, otherwise the command is absent and the test fails permanently. The path /usr/bin/pgrep was not replayed on Debian 13: on your node, confirm it with command -v pgrep, then run the test command by hand with echo $? (0 as long as HAProxy is running).
The vrrp_instance block describes the election. state BACKUP is written on both nodes: it is the priority that designates the master, and the manual requires an initial state other than MASTER for nopreempt to work. interface names the interface that carries the announcements; a wrong interface prevents the instance from starting. priority (from 1 to 255) breaks ties: the manual recommends 50 more for the master and priorities that are all different. advert_int is the interval of the announcements in seconds (1 by default). virtual_ipaddress lists the service address with its mask: the manual makes it an optional part of <IPADDR>[/<MASK>], but writing /24 avoids leaving the choice of the prefix to the tool, to check on your node with ip -o -4 addr show. track_script names the vrrp_script to apply: without it, the test never runs, because the manual specifies that a script is only launched if an instance watches it.
unicast_src_ip is the local address the announcements leave from, and unicast_peer the list of the addresses of the peer or peers, which receive the announcements instead of the multicast group. Multicast is the default mode: the announcements go to 224.0.0.18 and any node on the same layer 2 network hears them without knowing its neighbors. Unicast is the obligatory choice of a network that does not route multicast. In return, each node must know the address of the other: an error in unicast_peer severs the link and prepares a double master. The manual adds that it is impossible to run in unicast without a peer.
What changes on the second node, and what stays identical?
On proxy-b, only the priority and the two unicast addresses change; everything else must be identical to proxy-a, starting with the virtual_router_id. The virtual_router_id (from 1 to 255) is the identifier of the VRRP group: two nodes only see each other as peers if it is the same. With 77 on one side and 78 on the other, each node ignores the other's announcements, believes it is alone, elects its own master and takes the address: this is the split-brain (two machines that hold the address at the same time), described below, without any failure having occurred.
| Directive | proxy-a | proxy-b |
|---|---|---|
virtual_router_id | 77 | 77 (identical) |
virtual_ipaddress | 172.16.40.100/24 | 172.16.40.100/24 (identical) |
state | BACKUP | BACKUP (identical) |
priority | 150 | 100 (different) |
unicast_src_ip | 172.16.40.11 | 172.16.40.12 (swapped) |
unicast_peer | 172.16.40.12 | 172.16.40.11 (swapped) |
vrrp_script and track_script | identical | identical |
The vrrp_script block stays the same: if proxy-b has no test, a dead HAProxy on proxy-b is never detected, and the address stays there once it has taken it. The test command must also exist on proxy-b: if it is missing there, the test fails permanently. Before starting, keepalived -t -f /etc/keepalived/keepalived.conf (--config-test) checks the syntax, but not the values: a different virtual_router_id passes this check. After systemctl restart keepalived on both nodes, apply the check of the split-brain section, below: only one machine must hold the address.
Once healed, proxy-a takes the address back by preemption, as above, which creates a second outage. To avoid it, add nopreempt in vrrp_instance on both nodes; preempt_delay only delays the takeover.
Where do you look to read the state and diagnose a double holder?
Two commands are enough to know who holds the address and why: ip -o -4 addr show answers "who", and journalctl -u keepalived answers "why". Run ip -o -4 addr show dev ens18 on each node and look for the service address (here 172.16.40.100): one line on a single node is the healthy state. The log, read with journalctl -u keepalived --since "10 min ago" --no-pager, tells the transitions: Entering MASTER STATE or Entering BACKUP STATE lines for the election, and Entering FAULT STATE followed by sent 0 priority when a weight-0 vrrp_script fails (this last line only appears if keepalived runs with -D, --log-detail). The exact wording depends on the version, to check on your node.
Two holders are diagnosed in this order. First reflex: compare virtual_router_id, unicast_src_ip and unicast_peer on both nodes, because this is a very likely cause. Then, check that a firewall lets protocol 112 through (VRRP, which is neither TCP nor UDP) between the two addresses; this value comes from the definition of VRRP (RFC 5798) and not from the keepalived manual. Finally, reread the log of both nodes: a node whose state never changes while its peer stops is probably isolated. None of these steps replaces the measurement of the recovery described below.
The other setup failure is a node that never takes the address when it should, or whose state stays frozen. Then check that its vrrp_script does not fail permanently: run the test command by hand, then echo $?, which must show 0 as long as HAProxy is running. If the command is absent, the failure is constant: with a weight of 0 the node stays in FAULT, with a negative weight its priority stays lowered and it never takes the address back.
How do you recognize a split-brain before the first failover?
A split-brain is two machines holding the service address at the same time; at rest, only one of the two must hold it, and this is checked as soon as the setup is done. The nodes elect each other by exchanging announcements. If these announcements no longer get through (network partition, firewall blocking the protocol, wrong peer address), each node believes it is alone and takes the address. Both then answer on the same IP, and clients land on one or the other at random, with two diverging application states.
The check is simple and requires no failure: on each of the two machines, list the IPv4 addresses with ip -o -4 addr show and look for the service address. It must appear on one machine only. Zero holders, or two, is a setup defect to diagnose before any failover; two holders point first to announcements that do not circulate. Redo this check after each failover: the address must have changed machine and still be unique.
No configuration setting removes the risk of a partition: it is described in the operations record (the document for the people who run the service, which the lab asks you to write), with its consequence (the double master, that is, the split-brain) and how to detect it.
What must you keep identical on both nodes?
The two nodes must present the same service to the client: the same HAProxy configuration, a certificate valid for the same name and of the same trust, and the same access policy. Otherwise the failover becomes visible to the client, or worse, opens what the other node refused. A client that strictly verifies TLS must not see a different service depending on who answers.
Be careful about what this requirement exactly means, because the confusion is common. A strict client does not demand an identical file on the two machines: it checks that the certificate presented covers the requested name and that it is signed by an authority it trusts. Two distinct certificates for the same name, issued by the same authority, therefore pass the verification without the client seeing the slightest difference. This is even the usual case in production, where each node renews its own through ACME.
In the lab, the two entry points go further than this requirement and present the same fingerprint: the authority and the service certificate are generated once then copied, rather than generated twice. This is a choice of bench simplicity, which avoids having two expiry dates to follow during the exercise, and not a rule to transpose as is.
The same logic applies to the access policy. A private route rejected on edge must be rejected on edge2 too, after the failover: a policy that does not follow the service is an opening that goes unnoticed. The lab listens on bind :443 and not on the floating address, because HAProxy refuses to start on an address that the machine does not hold yet; yet the standby node does not hold the service address until it has taken over, and its HAProxy must start nonetheless. To listen on the floating address anyway, the HAProxy administration manual says to set the Linux kernel parameter (the sysctl) net.ipv4.ip_nonlocal_bind to 1; bind :443 avoids this setting.
A third category is more insidious: the state is not replicated. What an instance keeps in memory (counters, session tracking) and the decisions taken live on its admin socket, such as taking a server out of service (change the state at runtime), stay in that instance. After a failover, the other node starts again from its own configuration. Certificates, for their part, are managed as in HTTPS and ACME with HAProxy: each node must keep its own up to date.
How do you measure a recovery without touching the client?
You measure a recovery with a continuous probe that always queries the same service address, changing nothing, while you cause the failure; the duration of the interruption is read in the gap between the last answer and the first answer of the second node. The condition is strict: a failover that required an action from the client would not be one. A probe that you would restart on the other address would measure your own intervention.
The lab provides on poste.lab a probe named sonde-entree. It opens one strict TLS connection per second to the same address and writes one line per query: the instant, then the name of the machine that answered (the two application servers announce their name), or an error. In these lines, the failure shows up as a series of errors, and the recovery as the return of answers under the name of the other machine. The duration of the series is your recovery time: it is measured, not deduced from the configuration.
Run two tests, because they do not prove the same thing. First stop HAProxy on the holder node: this is the loss of a process, and the test that reveals the absence of a vrrp_script. Then reboot the whole machine: this is the loss of a host, which only two distinct hosts allow you to try.
Two points complete the measurement. First, a connection open at the time of the failure is cut: it dies with the process or the host that carried it, and the client must reconnect. It must not be confused with the case of a backend server declared DOWN by HAProxy. For the latter, the manual nowhere says in so many words that established connections survive; it is deduced from the inverse option, on-marked-down shutdown-sessions, which cuts them and of which the manual specifies "Actions are disabled by default" (the connections cut this way carry the termination code D). Second, capacity: if each node absorbs half of the traffic, the survivor receives everything. Each node must be able to carry the whole load, otherwise the failover turns a failure into a saturation.
What does this failover not protect?
This failover protects against the loss of an HAProxy process or of a host, not against a wrong configuration deployed on both sides, nor against the network partition, nor against the loss of the in-memory state. A configuration error pushed to both nodes fails over to a copy of the same failure. The operations record that the lab asks you to write covers three subjects, none of which can be read in the configuration: the double master that a partition produces, the unreplicated state, and what the failover does not cover at all (the upstream network, the backends, the configuration itself).
A failed shared backend is not repaired by a second proxy either: the health checks from server health remain necessary, behind each of the two nodes.
Practice
The lab starts from two HAProxy entry points already in place on edge.lab and edge2.lab, with the same configuration and the same certificate, and a keepalived installed but not configured. You deploy a failover compatible with the network provided, then you measure the loss of a process and of a host from poste.lab, without changing anything on the client side. The validation covers four checks: the two entry points meet the acceptance criteria (the starting acceptance test), the recovery is measured after the loss of a process then of a host, the two nodes present the same certificate fingerprint and the private route stays rejected after the failover, and the record describes what the failover does not protect.
Key points
The availability of an entry point is proven by the loss of a host, not only of a process.
- A failure domain is what goes down together: two instances on the same host share the host's failure.
- A VRRP floating address is only possible if the network accepts it; otherwise, a layer 4 upstream entry point or DNS, with their limits.
- Without a
vrrp_script, keepalived watches the machine and not the service; the test command must exist on each node, otherwise it fails permanently. - The
vrrp_scriptweight has two paths: at 0 (default), the failure puts the instance in the FAULT state; negative (the path taught here), it lowers the priority, which must drop below the peer's: with 150 against 100, a weight of -20 does not fail over, a weight of -60 does. - On both nodes, the
virtual_router_idis identical; only the priority and the unicast addresses differ. - At rest, only one machine holds the address: two holders signal a split-brain.
- Both nodes keep the same configuration and the same policy, and present a certificate valid for the same name and of the same trust, not necessarily the same file; the in-memory state is not replicated.
- A recovery is measured with a continuous probe that changes nothing on the client side, and each node must be able to carry the whole load.
Knowledge check
These questions check that you can predict what the failover does, and what it does not do, before measuring it.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Pangolin: expose a service without opening a port
A new full quiz with random questions
Work again only on the questions you missed