With Traefik 3.7.15, you cut a service over from v1 to v2 while limiting errors, thanks to a weight on each server of the same service and a health check on that service; however, none of this protects the entry point itself: two Traefik instances on the same machine go down together with it. This lesson is for anyone who already publishes a service behind Traefik and wonders "is it available?". It separates two questions that are often mixed up: the backend cutover (changing version without interrupting the clients) and the entry point failure (what happens to the service when Traefik, or its host, disappears). It also says plainly what a lab on a single machine cannot prove.
What you will learn
The lesson follows the order of a version change: shift, watch, then ask what happens if the proxy itself goes down.
- Shift traffic from
v1tov2with weights, and measure the errors. - Return to
v1with no intervention whenv2is removed from the rotation. - Tell the loss of a Traefik process from the loss of its host.
- Say what a lab on a single machine proves, and what it does not prove.
- Prevent a concurrent certificate rotation from leaving the pair mismatched.
Prerequisites
The lesson assumes you know how to write a Traefik route in a file and read an HTTP code. The lab provides the machine, the backends and the client.
- Traefik's model (entry point, router, service): Traefik: entrypoints, routers, services and middlewares.
- The dynamic file reloaded on the fly: the file provider.
- The health check, its parameters and the retry, which this lesson does not repeat: Control health checks, timeouts and long-lived streams with Traefik.
- Upgrading Traefik itself and rolling it back: Upgrade Traefik.
Which failures does a backend cutover handle, and which ones escape it?
A backend cutover handles the failures that happen behind Traefik; a failure of Traefik itself is covered by no backend setting. A backend is the application server that sits behind the proxy. A cutover is the gradual or abrupt move of traffic from one version to another. Both gestures are set in Traefik's configuration: server weights, health check.
A single point of failure is an element whose failure stops the whole service because nothing can replace it. The entry point is one as long as it is alone: all the traffic goes through it. You can have ten healthy, redundant backends, and the client sees nothing at all if the only Traefik is stopped.
In this lesson, an entry point means a Traefik process that receives client connections on its listening ports. Do not confuse it with the entryPoints section of the configuration, which declares those ports: when that section appears, it is named as such.
A failure domain is the set of everything that goes down at the same time when one element fails. The Traefik process forms one, the host that carries it forms another, wider one. The sister page on HAProxy sets the same frame; this lesson applies it to Traefik, covering the cutover first, then the failure of the entry point.
How do you shift traffic from v1 to v2 with weights?
You shift traffic by listing v1 and v2 as two servers of the same loadBalancer, each with a weight: the higher the weight, the more requests the server receives. The weight (weight) is an integer attached to a server. According to the Traefik 3.7 documentation, the default strategy, wrr (Weighted Round Robin), spreads requests across the servers according to these weights.
# FRAGMENT: service from the dynamic file, routers and tls not shownhttp: services: app: loadBalancer: servers: - url: http://pub-zds-app-v1:8080 weight: 3 - url: http://pub-zds-app-v2:8080 weight: 1At the start of the lab, the app service lists only v1, hard-wired. The block above replaces it with two servers; to move forward, you raise the weight of v2 and lower that of v1, then repeat. The dynamic file is hot-reloaded by the lab's two entry points: no restart is needed to change a weight. The weights 3 and 1 are an example, not a recommendation.
The lab test samples 40 requests and requires at least 5 responses from each version. A 10% share for v2 (weights 9 and 1) would give about 4 responses out of 40, below that threshold: this is a constraint of the test. Neither those weights nor the weights 3 and 1 were measured here. The same test tolerates at most 2 errors in the sample: the criterion is a bounded error, not a zero error (the last section of the lesson comes back to this).
The lab also imposes the form: two servers of the same service, and not the weighted type, which combines several named services (the sub-services) and gives each one a weight. The lab scenario states that this second type never removes an unhealthy sub-service in Traefik v3: that is too absolute. The 3.7 documentation describes a healthCheck specific to the weighted type: when one of its children is reported down, the weighted service ignores it. The 3.7.15 code (file pkg/server/service/service.go in the Traefik repository) only wires this signal if the weighted service itself declares a healthCheck, which requires all its descendants to have one, and the documentation only provides for it with the file provider. Without this healthCheck, a sick sub-service stays in the rotation. The lab's form remains the simplest for this lesson, but not the only one that removes a sick server. This point was read in the source, not replayed.
How do you return to v1 without human intervention?
The automatic rollback relies on a health check placed in the loadBalancer: it removes the failing server from the rotation, and traffic goes back to v1 on its own. A health check (healthCheck) periodically queries a route of each server; a rotation is the set of servers among which Traefik spreads requests. The probe's parameters (path, interval, timeout) and the length of its detection window are the subject of the lesson Health, timeouts and streams; they are not repeated here.
# FRAGMENT: to add under loadBalancer, next to servers healthCheck: path: /health interval: 2sIn the lab, you simulate the failure of v2 by deleting its marker /app/healthy with docker exec pub-zds-app-v2 rm /app/healthy. The backend then answers 503 on /health and 500 on its other routes. The test waits up to about fifteen attempts, one second apart, for v2 to disappear from the sample without any error, then restores the marker and checks that v2 receives traffic again. The health check must therefore do two things: remove a sick server, and reinstate it when it becomes healthy again.
A limit can be read in this criterion: the test waits for a clean state, it does not measure what happens during the detection window. Until the next pass of the health check has taken place, the sick v2 stays in the rotation and can produce errors. Reducing the interval shrinks the window without removing it, and a retry mechanism has its own risks, covered in the lesson on health and timeouts.
What is draining, and what does Traefik do when it stops?
Draining means ceasing to send new requests to a component while letting the ones in progress finish; Traefik does this for itself thanks to two delays set in the entryPoints section of its configuration. For a backend, draining amounts to taking it out of the rotation before stopping it. The health check does it after the fact, when it judges the server sick; it does not anticipate it when you decide to switch it off for an update. The lab does not measure this second case.
For Traefik itself, the 3.7 entryPoints reference documents transport.lifeCycle.graceTimeOut and transport.lifeCycle.requestAcceptGraceTimeout. The first is the time granted to active requests to finish before the graceful shutdown (the shutdown that lets ongoing work finish instead of cutting it off), 10 seconds by default; the documentation specifies that, during this delay, no new request is accepted.
The second, an acceptance delay, is the time during which Traefik keeps accepting requests before starting this graceful shutdown (0 seconds by default); the documentation explains it by the need to give an upstream load balancer (a device or service placed in front of Traefik) time to remove Traefik from its rotation.
# FRAGMENT: Traefik STATIC file, restart requiredentryPoints: websecure: address: ":443" transport: lifeCycle: requestAcceptGraceTimeout: 5s graceTimeOut: 15sThese values are an example for reading the option names, not a measurement: the lab does not test them. They concern the voluntary shutdown, for instance during an upgrade. They change nothing for an abrupt failure, where nobody had time to drain.
What happens to the service when an entry point goes down?
If a Traefik entry point goes down while a second one is running, the latter keeps answering; if both share the same machine, losing the machine takes them down together. The lab starts two twin entry points, pub-zds-entry-a and pub-zds-entry-b, which read the same dynamic configuration and serve the same certificate. They listen on two different ports of the same machine edge.lab. The measurement client, poste.lab, queries each one from the outside.
Look at the frame first: it surrounds the two entry points and the backends, because they share a host. The first dotted arrow only targets entry point A; the second targets the whole frame. That is the entire difference between the two failures.
The lab asks you to trigger them for real and to write your conclusion in /home/student/domaine-de-panne.yaml, with two keys, perte_processus and perte_hote, each taking the value survit or ne_survit_pas. Here, survit means "the other entry point still answers", not "the client noticed nothing". The instruction is not to guess: according to the scenario, claiming a resilience the lab does not have is worse than admitting its limit.
The test's two gestures are as follows. For the loss of a process, it stops the container pub-zds-entry-a with docker stop, then queries both ports from poste.lab: A must no longer answer 200, B must answer 200. For the loss of the host, it blocks the machine's two ports with an iptables rule (the Linux firewall), placed very early on the packet path: table raw, chain PREROUTING. According to the test's comments, an ordinary rule on the INPUT chain would not see this traffic, because Docker has already routed the packet to the container before it reaches that chain. The test removes the rule afterwards.
What does the lab prove about the entry point failure, and what does it not prove?
The lab proves that a second Traefik process keeps answering when the first one stops, and that blocking the two ports of the same host makes both entry points unreachable; it proves neither a client's failover, nor survival of a real machine failure. This is the limit you must be able to state, because it separates a demonstration from a guarantee. The lab scenario says it itself: the two entry points share a single failure domain, which the lab asks you to observe and not to work around.
| The lab demonstrates | The lab does not demonstrate |
|---|---|
| Entry point B answers when process A is stopped | That a client moves from A to B without being told |
Two blocked ports make A and B unreachable from poste.lab | That a machine that is really switched off, or a full disk, behaves the same way |
| The two failures give two different results | That a single address can move to another machine |
A stops answering 200 when its process is stopped | The real recovery time of a failover mechanism |
Two points deserve emphasis. First, the test client queries A then B, each on its own port: no mechanism in the lab moves it from one to the other by itself. Saying "B answers" therefore does not say "the client is served". Second, blocking two ports with a network rule simulates unreachability seen from the client, not the failure of a machine: the processes are still running, and it is the test itself that blocks the two ports. That the two entry points share their fate therefore comes from the construction of the lab (a single machine), not from a measurement; the test measures no recovery. It also compares your file to two fixed values: it does not judge your reasoning.
How do you survive the loss of a host?
To survive the loss of a host, you need at least two distinct hosts, which do not in turn share a single element (hypervisor, power supply, network), and a way for clients to reach the second one without changing anything on their side. Making containers redundant on a single machine does not remove the failure of that machine: that is the mistake the lab makes you touch. Three paths exist, each with a limit.
A virtual address (or floating address, VIP) is a service address that is wired to no machine in particular: the elected machine carries it, and if it goes down, another takes it over. The VRRP protocol and the keepalived daemon are the classic path; the lesson Test the availability of the HAProxy entry point teaches the full mechanism, including its main trap (monitoring the service and not the machine). It was not replayed with Traefik here: the principle is not specific to a proxy, but it is a point to validate on your network.
The other two paths are an upstream load balancer (a device or service that distributes connections in front of the two Traefik instances, and which itself becomes the point to protect) and DNS with several addresses, whose removal of a dead address depends on the clients' cache. Each of these three paths moves the problem without abolishing it: you must know what remains single.
A container orchestrator such as Kubernetes changes the frame: the page Traefik on Kubernetes (in French) recommends deploying several replicas (copies of the same program) for high availability, which this lab does not cover.
How do you share a certificate between two entry points without corrupting it?
Two entry points that serve the same certificate must read a consistent pair, and the tool that renews it must hold up when two runs happen at the same time. In the lab, the two Traefik instances mount the same certificate directory read-only, and a script, /opt/pub-zds/outils/tourner-certificat.sh, regenerates it. Written on the assumption that it never runs twice at once, this script drops the certificate then the key, or the reverse, with a pause between the two: two simultaneous runs can leave a certificate from one and a key from the other. A mismatched pair, whose certificate and key do not correspond, cannot serve valid TLS.
The test launches two runs in parallel, several times, then requires the certificate and the key to be readable and their public keys to coincide. Finally it requires the two entry points to still serve valid TLS. Fixing the script is up to you. The reasoning to follow is this: renaming a file is atomic (the operation happens in one block, with no visible intermediate state), but renaming two files one after the other does not form a single operation, because another run can slip in between. Serialization, which makes runs go one after the other thanks to a lock (the flock tool, documented by its manual), is the other path cited by the lab. Choosing and verifying is your job. A lock only orders the writers among themselves: it does not prevent a reader from landing between two writes.
In the real world, sharing a certificate between instances gets complicated with ACME, the protocol by which Let's Encrypt issues and renews certificates automatically. The ACME page of the Traefik 3.7 documentation states, in the subsection devoted to Let's Encrypt with the Kubernetes Ingress provider, that it is not possible to run several instances with Let's Encrypt enabled, because there is no guarantee that the right instance receives the validation challenge, and recommends Traefik Enterprise for that. This subsection is written for version 2.0 and has not been checked against the behavior of 3.7. The lab's script is a local, offline representation of this sharing, not a test of ACME. For managing the certificates of a single instance, see Traefik and automatic HTTPS.
Does a test with zero errors prove zero interruption?
No: a bounded test only proves that, on this sample, at this moment and with this load, the number of errors stayed within the tolerated limit (2 at most here); it does not guarantee the absence of interruption under another load. The three measurements of this lesson each have their bound. The 40-request sample tolerates 2 errors; the removal of v2 is judged on a clean state reached after a few seconds; the host failure is a network rule on two ports.
What remains invisible: a long request cut by a shutdown, a heavier load that lengthens the health check's detection window, a connection already open at the moment of the failure (it dies with the process that carried it, the client has to reconnect). A test that passes is limited evidence, which you must name as such in your report. Writing "tested on 40 requests, 2 errors tolerated" tells the truth; writing "zero downtime" does not.
Put it into practice
The lab starts from two twin Traefik entry points on edge.lab, a v1 backend in production and a healthy v2 candidate. You build a gradual path from v1 to v2 with two weighted servers, you add the health check that removes v2 when it is unhealthy, you trigger the two failures from poste.lab to write an honest conclusion file, then you fix the certificate rotation tool. The validation measures the real distribution of traffic, the removal then reinstatement of v2, your conclusion file compared with the two failures that the test triggers itself, and the consistency of the certificate and key pair after concurrent rotations.
Key points
The availability of a service behind Traefik is judged in two places: behind the proxy, and on the proxy itself.
- A backend cutover is set by weights on the servers of the same
loadBalancer; a health check removes and reinstates them. - The lab's distribution criterion is a bounded error (2 out of 40 at most), not zero; the health check's detection window is not measured.
- Draining applies to a voluntary shutdown:
graceTimeOutandrequestAcceptGraceTimeoutare its settings, not tested here. - A lone entry point is a single point of failure; two on the same host share a failure domain.
- The lab proves that B answers when process A is stopped, and that both entry points become unreachable when their ports are blocked; it proves neither client failover nor a real machine failure.
- Surviving the loss of a host requires two hosts and a virtual address, an upstream load balancer or DNS, each with its limit.
- A certificate rotation tool must hold up to two simultaneous runs: serialize or make the publication atomic, do not assume.
Knowledge check
These questions check that you can predict what each setting covers, and what it leaves uncovered.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Level 1 assessment: publish two services over HTTPS and survive a restart
A new full quiz with random questions
Work again only on the questions you missed