To update an application behind HAProxy without cutting anyone off, reload the configuration instead of restarting the service, never remove the generation you have just replaced, and keep a way back. This lesson is for someone who already runs an HAProxy 3.4 and must deliver a new version of their application, then take it back if it misbehaves. It explains the master-worker pair, the delivery rule that follows from it and how to prove that a cutover cut nothing. It covers neither the admin socket nor the live state of the servers.
What you will learn
The lesson goes from the mechanism to the proof, through the delivery rule.
- Describe what a reload does to the master process and its workers.
- State the rule: a delivery removes the generation replaced by the previous delivery, never the one it replaces.
- Justify why the master is never replaced, and read its PID to prove it.
- Deliver a reversible cutover between two generations, with its way back.
- Interpret what the validation measures, and what it does not see.
Prerequisites
The lesson assumes a running HAProxy 3.4 in front of an application, and an account that can use sudo. The facts were established on HAProxy 3.4.7, the lab version, on Debian 13.
- Install HAProxy, read its sections and validate a configuration with
haproxy -c: Install HAProxy and read its configuration sections. - Remove a server without cutting what it serves, and know what a long stream is: Choose the load-balancing algorithm and measure health checks.
- Reload and read a systemd unit: systemctl: list, start and diagnose Linux services.
What does a HAProxy reload really do?
A reload does not replace the master process: it makes it start new workers, and lets the old ones live until the connections they hold are finished. In master-worker mode (activated by the -W option), HAProxy is made of two roles. The master supervises and restarts the workers; the workers carry the configuration, handle the traffic and hold the listening sockets (a network listening point for each open port). On a reload, the master retrieves these sockets from the current worker through a file descriptor transfer (one process hands another a socket that is already open), then hands them to the new ones: the port is therefore never closed again from one generation to the next. A systemctl reload haproxy sends the master the signal that asks it to reread the configuration and launch a new generation of workers.
The old workers no longer accept new connections: those go to the new ones. But they keep serving the ones they already held, and stop only once the last one has ended. The two generations of workers therefore coexist, sometimes for a few minutes. This is exactly what makes a reload without interruption possible.
The diagram follows a long stream through a reload. Look at steps ① and ③: the same client first talks to the old workers, then its new requests reach the new ones, while its open stream ends on the old ones.
The two generations of workers can be seen on the machine. The workers are child processes of the master, and the master PID (its process identifier) is the one that systemd knows.
maitre=$(systemctl show -p MainPID --value haproxy)pgrep -P "$maitre"The first line reads the master PID, the second lists its children. Right after a reload, if a long stream is still open, you see more workers than before: the old ones and the new ones. When the last stream closes, the old ones disappear and the list returns to its original size. This is also the exact measurement the lab validation makes: it compares the set of children before and after, and requires that at least one new worker has appeared.
The master also exposes a command socket, opened by the -S option, which accepts the show proc command to list the processes. It is described, with its rights, in Change HAProxy's state at runtime with limited rights: that lesson is about what you change inside a running HAProxy, this one about what you change around it.
Why does a delivery not remove the generation it replaces?
Because the old workers still talk to the generation they knew: stopping it would cut the streams they serve. A generation here means one version of the application with its own process and its own port (1.4.2 on port 18211, 1.5.0 on 18212 in the lab). At the moment of the reload, the configuration points to the new generation, but the old workers have kept the previous configuration.
The trap is to deliver "cleanly": start 1.5.0, reload, then stop 1.4.2 since it no longer serves anything. Nothing breaks on a short request. A long stream opened before the cutover, however, is served by an old worker that talks to 1.4.2: stopping it cuts the stream in the middle. This defect only shows in the middle of a cutover, which explains why it passes cold tests.
The rule is therefore: a delivery removes the generation that had been replaced by the previous delivery, not the one it replaces. By then, the old workers that talked to that generation have had all the time to finish. The generation being replaced stays alive, and doubles as a free rollback.
| Delivery | In service before | In service after | Removed by this delivery |
|---|---|---|---|
| None (start) | 1.4.2 | 1.4.2 | nothing |
livrer 1.5.0 | 1.4.2 | 1.5.0 | nothing (1.4.2 stays alive) |
| Next delivery, say 1.6.0 | 1.5.0 | 1.6.0 | 1.4.2, replaced by the delivery before |
Two generations only exist in the lab: the third row of the table is an example that the lab does not play. Remember the mechanism: you always keep the generation in service and the one it has just replaced, and you remove the one before.
Why must you never replace the master process?
Because only the master hands the listening sockets from one generation of workers to the next: replacing it takes the whole service with it, closes the port while another process reopens it, and no client-side measurement makes up for a second during which nothing listens. A systemctl restart haproxy stops the service then starts it again: master and workers disappear together, taking the sockets with them, the successor has another PID, and port 80 is closed in between. It is the continuity of the master that makes the reload transparent, not the fact that it would listen itself.
A client probe that fires one request every 200 ms may well see nothing of this gap. The proof is therefore not made by the traffic but on the machine: you read the master PID before and after the cutover, and a different PID signs a restart disguised as a reload.
avant=$(systemctl show -p MainPID --value haproxy)sudo systemctl reload haproxyapres=$(systemctl show -p MainPID --value haproxy)[ "$avant" = "$apres" ] && echo "maître inchangé" || echo "maître remplacé"The block compares the two PIDs. If you read "maître inchangé" (master unchanged), the port was never closed. This is the measurement the validation makes, with systemctl show -p MainPID --value haproxy, before and after running your delivery. A reload keeps the master; a restart, and any script that goes through it, changes it.
How do you deliver a reversible cutover between two generations?
Rewrite a single line in a single file, validate, reload, and stop nothing that the old workers still use. The lab isolates what changes in /etc/haproxy/service.d/generation.cfg, a file included by HAProxy in addition to haproxy.cfg (two -f options, added by an override of the unit). The line that changes is the server line:
backend applicatif option httpchk GET /pret http-check expect status 200 server courante 127.0.0.1:18211 check inter 1s fall 3 rise 2Switching to 1.5.0 amounts to replacing port 18211 with 18212. Nothing else is involved, which makes the delivery reviewable and the way back identical to the way there. The lab expects two executables, livrer <version> (deliver) and revenir (go back), in the language of your choice: the validation calls them, it does not read them.
The steps of livrer, in this order, are always the same:
- Start the requested generation and wait for it to answer, before showing it to HAProxy. A systemd template unit
pub-maj-app@<version>(a single unit file, where what follows@names the version) starts it without editing anything:systemctl enable --now. - Rewrite the line in the file, after keeping a copy.
- Validate with
haproxy -con both files, and restore the copy if the validation fails: you never reload an invalid configuration. - Reload with
systemctl reload haproxy, neverrestart. - Remove the generation replaced by the previous delivery, and note the one you have just replaced for the next time.
Here is a sketch of one possible logic, to adapt: the lab imposes neither a language nor a structure, and this version is not the lab's solution replayed as is. It keeps in two small files the generation in service and the one it replaced.
#!/usr/bin/env bashset -euo pipefailcible="$1"etat=/var/lib/pub-haproxy-mise-a-jourgen=/etc/haproxy/service.d/generation.cfgdeclare -A port=( [1.4.2]=18211 [1.5.0]=18212 )
install -d "$etat"courante=$(cat "$etat/courante" 2>/dev/null || echo 1.4.2)precedente=$(cat "$etat/precedente" 2>/dev/null || true)
systemctl enable --now "pub-maj-app@${cible}.service"until curl -sf "http://127.0.0.1:${port[$cible]}/pret" >/dev/null; do sleep 1; done
cp "$gen" "$gen.avant"sed -i "s/127.0.0.1:[0-9]*/127.0.0.1:${port[$cible]}/" "$gen"if ! haproxy -c -f /etc/haproxy/haproxy.cfg -f "$gen"; then mv "$gen.avant" "$gen"; exit 1fisystemctl reload haproxy
if [ -n "$precedente" ] && [ "$precedente" != "$cible" ] \ && [ "$precedente" != "$courante" ]; then systemctl disable --now "pub-maj-app@${precedente}.service"fiif [ "$courante" != "$cible" ]; then systemctl disable "pub-maj-app@${courante}.service"fiecho "$courante" > "$etat/precedente"echo "$cible" > "$etat/courante"Reread the end of the script. The generation removed by disable --now is precedente, read before being overwritten: it is the rule of the previous section, in one condition. The replaced generation only receives disable, without --now: it keeps running for the old workers, but will not restart at the next boot of the machine. At reboot, what starts is what HAProxy's file designates, nothing more.
One writing detail deserves a detour, because it breaks delivery scripts much more often than you would think. The last condition is an if, and not a [ … ] && command that would say the same thing more briefly. Under set -e, this shortened form is dangerous in one precise place: the last one in a script or in a function. A false test stops nothing at that moment, because set -e ignores the failure of a command placed to the left of an &&; but the test's non-zero code becomes that of the script or of the function, and that is what stops the caller. Measured on bash 5.2.21: followed by another instruction, the line lets the script carry on and exit 0; as the last instruction of a function, it makes the script exit 1, and the line placed after the call is never run. The failure is then silent: nothing really failed, and yet the script stopped.
How do you roll back without cutting the service?
The way back is a cutover like any other: you replay it towards the previous generation, with the same validation, the same reload and the same availability requirement. A rollback that cuts the service is not one, because you run it precisely when something is already going wrong, and that is the moment when one more cut costs the most. The lab validation treats revenir as a critical check, by the way.
With the sketch of the previous section, revenir fits in a few lines: it reads precedente and calls livrer again with it.
#!/usr/bin/env bashset -euo pipefailetat=/var/lib/pub-haproxy-mise-a-jourexec "$(dirname "$0")/livrer" "$(cat "$etat/precedente")"The way back holds no surprise because the previous generation is still alive, for the reason of the rule: no delivery removed it. The way back is therefore measured like the way there. Note also that the condition [ "$precedente" != "$cible" ] prevents livrer from removing the generation you are going back to, which is precisely the previous one read at the start: a rollback never destroys its own point of arrival.
That leaves the case of the machine that reboots. A "1.5.0 in service" state that lived only in the memory of the processes disappears at reboot, whereas the file that HAProxy reads at startup may say something else. The lab tests this consistency last: it reboots the machine, checks that 1.4.2 is served again, then replays a complete delivery and its rollback. The cycle must stay operational after a reboot, not just the service.
What does the lab validation prove, and what does it leave aside?
The validation proves that a cutover played under load cuts neither the port, nor an established stream, nor more than one short request; it does not prove that you respected the removal rule. This is a point to read without indulgence, because a proof wider than what it measures is the first cause of misplaced confidence.
For each cutover, way there and way back, the validation:
- opens a long stream from the client machine, and waits for its first lines;
- starts a continuous probe of short requests;
- reads the master PID and the list of workers, then runs your command;
- notes that the master PID is unchanged and that a new worker has appeared;
- requires at most one failed short request;
- requires that the long stream keeps receiving and runs to its end, without error, and without changing generation along the way;
- finally checks that the version served is the expected one.
Notice what this list does not contain: no line checks that the previous generation is stopped or kept. The removal rule is measured indirectly: if you stop the replaced generation, the long stream does not run to its end and the check fails. On the other hand, a delivery that lets too many generations live, or that never removes the oldest one, makes no test fail. That half of the rule is up to you.
Beyond these checks, the journal is read by a human. The test only verifies that the file exists, that it exceeds 400 characters and that it speaks of the stream or of established connections, and of the rollback. The rest, what you measured during the cutover and how to go back, is what a reviewer will judge.
The measurement has a limit that the training does not hide. A reload without interruption says nothing about a machine failure: a reboot or a host loss cuts every stream, old workers included. No cut on a reload does not mean no cut on a failure. Diagnosing unexpected behavior is covered in Diagnose HAProxy without masking the cause.
One last operating point: an old worker stops only when its connections end. With a timeout client (the inactivity tolerated on the client side) of 180 seconds, as in the lab, an idle connection can hold it for a long time. HAProxy can bound this duration with the hard-stop-after directive of the global section. The 3.4 manual defines it as "the maximum time allowed to perform a clean soft-stop", that is, the duration beyond which the instance exits even if open connections remain. It therefore trades a cleaner end for the old workers against the cutting of the streams that exceed the delay.
Practice
The lab hands you two generations of the same application, 1.4.2 in service and 1.5.0 stopped, and no way to go from one to the other. You write livrer and revenir (two executables, in the language of your choice), then a journal.md. The validation plays your executables while it measures: a long stream opened before the cutover, a probe of short requests, the master PID and the workers before and after, for the way there then the way back. It also checks the starting state, reads your journal, then reboots the machine and replays the complete cycle.
Key points
- A reload keeps the master and makes it start new workers; the old ones finish their connections before stopping, and the two generations coexist for a few minutes.
- A delivery does not remove the generation it replaces: the old workers still talk to it. It removes the one replaced by the previous delivery.
- The workers hold the listening sockets, and the master hands them from one generation to the next: replacing it takes the service with it and closes the port, which a client does not always see. You prove it by its PID, read before and after with
systemctl show -p MainPID --value haproxy. - A cutover is played in a few steps: start the generation, rewrite one line, validate with
haproxy -c,systemctl reload haproxy, then remove the generation before. - The rollback is a cutover like any other, with the same measurement; it holds no surprise because the previous generation is still alive.
- What the lab measures: port, established stream, one short request at most, master PID, new worker, rollback, reboot. It does not check that the oldest generation was actually removed.
- No cut on a reload does not mean no cut on a host failure;
hard-stop-afterbounds the life of the old workers, at the price of cutting streams longer than the delay.
Knowledge check
Check that the essentials of this guide are acquired. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Test the availability of the HAProxy entry point
A new full quiz with random questions
Work again only on the questions you missed