For a Traefik 3.7.15 to stay reliable in front of a slow, dead or chatty backend, bound the backend connection and response in a server transport, lengthen the entrypoint's write timeout for streams, add a health check and do not let a retry replay a write. A health check is a periodic verification of each backend; a retry is an automatic new attempt, which can count an action twice when it is badly framed. This lesson is for anyone who already publishes a service behind Traefik and finds out that a setting that looks reasonable can produce silent failures: an event stream cut at a fixed time, a duplicate the client never sees. It applies to Traefik the principles of the lesson on long-lived streams and rests on measurements replayed on version 3.7.15.
What you will learn
The lesson goes from the placement of timeouts to the proof, setting by setting.
- Place each timeout on the leg it watches: client to Traefik, or Traefik to backend.
- Bound a silent backend without breaking a slow but legitimate response.
- Explain why an active SSE stream is cut by the write timeout.
- Tune a health check and predict the time it takes to act.
- Prevent a lost response from making an action count twice.
- Prove each setting with a measurement taken from a client.
Prerequisites
The lesson assumes you know how to write a Traefik route and read an HTTP code. The lab provides the machine, the four backends and the measurement clients.
- Traefik's model (entrypoint, router, service): Traefik: entrypoints, routers, services and middlewares.
- Publishing several instances of the same service and the idea of a health check: Traefik with Docker.
- The concept of a long-lived stream, an idle timeout and a heartbeat: Recognize the needs of long-lived streams and application protocols.
- Writing a dynamic routes file and reloading it: the file provider.
Where does Traefik set its time bounds?
Traefik watches two legs with two families of settings: the client-to-Traefik leg is set on the entrypoint, the Traefik-to-backend leg is set in a server transport. A backend is the application server behind the proxy. A timeout is a duration after which Traefik gives up and answers with an error itself, instead of waiting forever.
The server transport (serversTransport) is a dynamic configuration object that a service cites by name. It carries the forwardingTimeouts: dialTimeout to establish the network connection (TCP) to the backend, and responseHeaderTimeout to wait for the start of the response. The entrypoint, for its part, is Traefik's listening port. Its configuration is static, read at startup, and carries transport.respondingTimeouts with, notably, writeTimeout, whose default value is 0s (no timeout) according to the Traefik 3.7 documentation. The other two timeouts in this block, readTimeout (reading the whole request, 60s by default) and idleTimeout (idle connection between two requests, 180s by default), are not covered here.
This difference in nature has a practical consequence. The dynamic file is hot-reloaded by Traefik. The static file is not: changing writeTimeout requires restarting Traefik, which the lab recalls in its README. The page Troubleshoot a Traefik entry point already cites the default values of two of these timeouts: 30 seconds for dialTimeout and no timeout for responseHeaderTimeout.
Look at the diagram from left to right: the two transport bounds only cover their own step, whereas writeTimeout covers the whole line. This detail decides everything that follows, including the fact that the two families of settings must be chosen together.
How do you bound a backend that never answers without cutting legitimate slowness?
Set dialTimeout and responseHeaderTimeout in the server transport, above the slowest legitimate response and below what the client tolerates. Without responseHeaderTimeout, a backend that accepts the connection and then goes silent leaves the client waiting indefinitely: Traefik has no reason to give up. The lab starts from a transport named lent, set to 30 seconds on both timeouts: a user in front of a silent backend therefore waits up to half a minute before seeing an error.
# FRAGMENT: goes in the dynamic file; services and routers not shownhttp: serversTransports: borne: forwardingTimeouts: dialTimeout: 3s responseHeaderTimeout: 5s services: api: loadBalancer: serversTransport: borneThe block declares a transport named borne, then the api service cites it in serversTransport. The values of 3 and 5 seconds are those of the measurement below, not a recommendation: choose them against your slowest legitimate response, with a margin. A transport applies to the services that cite it, and only to them: in the lab, the service that serves the SSE stream cites none and is therefore not affected.
Measured on Traefik 3.7.15, with this transport: a legitimate response of 1.5 seconds returns 200 in 1.50 s; a sleeping backend returns 504 in 5.00 s, exactly the bound; a response that takes 7 seconds also returns 504 in 5.00 s. The 504 (Gateway Timeout) is the code Traefik produces itself when the backend has not answered in time.
Remember the scope of responseHeaderTimeout: once the request has been entirely sent to the backend, it measures the time until the start of the response, that is, its headers. A backend that sends its headers right away and then trickles the body for minutes is therefore not targeted by it: it is not an idle timeout on the body (a timeout that restarts from zero at every byte received). The case of a stream served behind a short responseHeaderTimeout was not measured here.
Why does the entrypoint's write timeout cut an active SSE stream?
writeTimeout is an absolute deadline, a total duration granted to writing a response, not an idle timeout: an SSE stream that emits one event per second is cut all the same, when the deadline arrives. An SSE (Server-Sent Events) is an HTTP response that does not end and in which the server writes events over time. The deadline is the moment when the granted duration has elapsed. The lesson on long-lived streams states that a read timeout restarts at every byte, and that a heartbeat (a small message sent at regular intervals to show that the link is alive) does not protect against a total-duration timeout. The first rule is true of idle timeouts, but writeTimeout is not one, and that is the most misleading difference in this lesson: the measurement below is the one the lesson on long-lived streams cites.
Here is what was measured, with a backend that emits one event per second, with no idleness at all.
Entrypoint writeTimeout | Observation window | Events received |
|---|---|---|
3s (the lab's starting setting) | 8 s | 3 |
20s | 25 s | 20 |
0s (no timeout) | 8 s | 8 |
In the first two rows, the number of events is the number of seconds elapsed up to the deadline: the stream is closed at writeTimeout, even though it was never silent. A heartbeat placed in the stream changes nothing, because it does not reset the clock. In the measurement, only the value 0s, which removes the protection, let the stream live. The lab's scenario speaks of a cut "after a short period of inactivity"; the measurement shows a cut at a fixed duration.
This single clock has a consequence for the transport bounds. writeTimeout also covers the wait for the backend, since the clock runs from the end of reading the request headers to the end of writing the response (that is the definition in Traefik's documentation). With a writeTimeout of 3 seconds and a transport that gives up at 5, a silent backend therefore returns no 504 at all: the client sees the connection closed with no response (curl prints the code 000 after 5.00 s). With these same settings, a legitimate response of 4 seconds suffers the same fate, whereas the 2-second one goes through. The transport bounds no longer produce their 504 as soon as writeTimeout is shorter: they must be set together, with a writeTimeout greater than the longest wait you accept.
# FRAGMENT: Traefik's STATIC file, a restart is requiredentryPoints: web: address: ":80" transport: respondingTimeouts: writeTimeout: 0sThe block removes the bound for the web entrypoint. It is the setting that let all eight events out of eight through in the measurement, but it also removes the bound that protects, among other things, against a client that reads too slowly (that effect was not measured). Setting a high value rather than zero keeps a protection, at the price of a stream that closes at the deadline: the client must then reconnect. This trade-off is a judgment call, not a rule: it depends on the lifetime you expect from a stream. Since writeTimeout is set on the entrypoint, it applies to all the routes that use it; moving the streams to a dedicated entrypoint is a path to consider, which the lab does not implement.
The WebSocket, an HTTP connection that switches to a channel where client and server write to each other at any moment, behaves differently. After the 101 response (the one that accepts the protocol switch), the connection becomes a tunnel: Go's HTTP server, on which Traefik is built, clears the connection's deadlines at the moment Traefik takes it over (an operation Go calls hijack). Measured with a writeTimeout of 3 seconds, a WebSocket client that waits 6 seconds before sending its message does receive its echo. In this measurement, writeTimeout did not cut the tunnel. Do not generalize to other entrypoint timeouts, which this lesson did not measure.
What does the health check measure, and how fast does it remove a backend?
The health check (healthCheck) periodically queries a route of each backend and removes from the rotation (the set of backends among which Traefik spreads requests) the one that does not answer correctly, but not before its next pass. Without a health check, a sick backend stays in the rotation and receives its share of traffic. In the lab, one of the two API backends becomes sick when a file is deleted: with two backends sharing the load equally, 10 requests out of 20 fail, even though its peer is perfectly healthy.
# FRAGMENT: goes under loadBalancer, next to servershealthCheck: path: /health interval: 1s timeout: 1sThe check sends a request to path, every interval, and waits for the response for timeout at most. With no status setting, any code from 200 to 399 counts as "healthy" (the documentation says "between 2XX and 3XX"); the 503 of the lab's sick backend is recorded by Traefik as a failure (checking HTTP health: received error status code: 503). The default interval is 30 seconds and the default timeout is 5 seconds (Traefik 3.7 documentation); the lab's measurement agrees: two successive failures appear in the log 30 seconds apart. The default setting therefore leaves a sick backend in the rotation for up to half a minute.
The detection window is measured by counting the failures right after the outage: 20 short requests are sent at a given moment after the backend is lost, and the failing ones are counted.
| Health check setting | Moment of the measurement | Failing requests out of 20 |
|---|---|---|
| no health check | 2 s after the outage | 10 |
interval: 10s | 1 s after the outage | 10 |
interval: 10s | 12 s after the outage | 0 |
interval: 1s, timeout: 1s | 2 s after the outage | 0 |
Read the 10-second row: as long as the next pass has not happened, the backend is still in the rotation. A shorter interval shrinks the window, at the cost of more probe requests to each backend; that is a load trade-off, not an absolute rule. The lesson Traefik with Docker explains the difference with the unhealthy state that Docker reports, and recalls that a health check alone does not remove the error during the window.
Two limits to keep in mind. The check only sees the health route, never the business routes: a backend whose /health answers 200 while its other routes fail stays in the rotation. It does not restart the backend either: it only changes the routing. In the lab, the health routes and the business routes depend on the same file, which makes the measurement readable but proves nothing about a real backend.
How do you stop a retry from counting an action twice?
A retry middleware (retry) replays a request when the backend has not answered, and it must not replay an action that writes, unless that action is idempotent. An idempotent operation gives the same final state whether it is run once or ten times. A read is. A POST that increments a counter is not, and replaying it produces a duplicate. The retry (new attempt) is useful against a passing failure, but it does not know whether the write already happened before the cut.
The lab recreates the trap. The action backend increments a counter, then closes the connection without answering: the write is done, the response is lost. With the lab's starting setting, which contains retryNonIdempotentMethod: true, Traefik takes the silence for a failure and replays the POST. Measured: the counter ends at 2 for a single action, and the client receives a 200. The duplicate is therefore invisible to it: only the backend's counter reveals it.
# FRAGMENT: retry middleware without the flag that replays writeshttp: middlewares: retry-lecture: retry: attempts: 3With this middleware, which no longer contains retryNonIdempotentMethod, the same experiment returns a 502 to the client and leaves the counter at 1: the POST is not replayed. The client sees an honest error, instead of a success that hides a duplicate. Another way is not to attach the middleware to the action router: it then replays nothing on that route. The choice depends on whether other routes of the same router deserve a retry, which the lab does not measure.
A 502 is still a half-answer, though: the client does not know any better than Traefik whether the action took place. For it to be able to replay safely, it is the application that must recognize an already processed request, for example with an idempotency key supplied by the client. That belongs to the service, not to the proxy, just like the heartbeat of a stream, which the lesson on long-lived streams assigns to the service.
How do you prove each setting from a client?
You prove a timeout setting by timing a request that must fail or succeed, not by rereading the configuration file. The code alone is not enough: a 000 from curl means that no HTTP response was received (connection closed, or --max-time elapsed), and it is a different failure from a 504. The -w option of curl prints both the code and the duration.
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 40 \ -H 'Host: api.flux.lab.test' http://127.0.0.1/bloqueThe command sends the request to the route that never answers, with the Host header (the site name requested, which the router uses to recognize the route). With the bounded transport described above and a wide enough writeTimeout, the output should show 504 and a duration close to 5. A 000 obtained before the --max-time deadline, on the other hand, very probably betrays a cut by the entrypoint: Traefik could no longer write its response (this is a deduction from the semantics of writeTimeout, only the 000 is measured). The access log, described in Observing Traefik, gives the same information in its Duration field.
| What you want to prove | Measurement | Reading |
|---|---|---|
| Legitimate slowness goes through | /lent?ms=1500, --max-time 4 | 200 in about 1.5 s |
| A silent backend fails fast | /bloque, --max-time 10 | 504 (or 502 if the backend cuts the connection) before 10 s |
| A stream lives beyond the deadline | curl -N --max-time 5 on /events, count the data: lines | about one event per second |
| A sick backend is removed | 20 short requests after the outage | 0 failures |
A POST is not replayed | GET /compteur after a lost action | 1 |
For the stream row, curl -N disables client-side buffering: without it, it is curl that would delay the display and wrongly blame Traefik. A measurement that passes once does not prove robustness: to judge a writeTimeout, leave the stream open longer than the deadline and count.
What does the lab validate, and what does it leave to your judgment?
The lab validates behaviors measured from the client, not precise directives. Several checks are run on the lab machine: a legitimate wait of 1.5 seconds succeeds in under 4 seconds; a silent backend fails with a 502 or a 504 in under 10 seconds; the SSE stream delivers enough events over a five-second window; the WebSocket returns its echo; none of the 20 short requests fails while a backend is sick; and a POST whose response is lost counts only once.
What the lab does not measure matters just as much. It requires no precise value for any timeout, does not check that you keep a protection rather than removing writeTimeout, and does not wait for the WebSocket after a period of inactivity: its client is launched with no wait. It says nothing either about the best organization between a single entrypoint and an entrypoint dedicated to streams. That part is yours: reread your values by asking whether they would hold against your real traffic, not just against the test.
Hands-on
The lab gives you a Traefik with four wrong settings: an overly generous transport, a write timeout too short for a stream, an absence of health check and a retry that wraps a non-idempotent action. You fix the routes file and the static file, restart Traefik when necessary, then let the validation send its requests from a client. It checks the behaviors described in the previous section, not the shape of your configuration.
Key points
A reliable Traefik is tuned leg by leg and proven by measurement.
- The Traefik-to-backend bounds are set in a
serversTransport; those of the client-to-Traefik leg on the entrypoint, in static configuration. responseHeaderTimeoutmeasures the wait for the headers, not for the body; a silent backend is bounded with it, and returns a504at the deadline.writeTimeoutis an absolute deadline, not an idle timeout: an active SSE stream is cut at the deadline, and the value0sremoves the bound.- A
writeTimeoutshorter than the transport bounds prevents Traefik from producing its504: the client receives a closed connection. - A health check only acts at its next pass: the default interval is 30 seconds, and the interval sets the detection window. It only sees its health route and restarts nothing.
retryNonIdempotentMethod: truereplays aPOSTwhose response was lost: the counter rises to 2 while the client receives a200. Without that flag, the client receives a502and the counter stays at 1.- In the measurement taken, a WebSocket is not cut by
writeTimeoutafter the101response, which does not generalize to the other entrypoint timeouts. - A setting is proven with
curl -w '%{http_code} %{time_total}s', not by rereading the configuration.
Knowledge check
These questions cover the scope of each timeout, the detection window of a health check and the replay of a write.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Protecting the Traefik entry point
A new full quiz with random questions
Work again only on the questions you missed