For HAProxy 3.4 to serve an API with long streams without cutting them or replaying a write, give each slow phase its own time bound, set the tolerated silence to the application's heartbeat, stick sessions by cookie and limit retries to requests that never reached the server. This lesson is for readers who already publish a service behind HAProxy and want, at three in the morning, the time of an error to tell which phase is slow. It does not repeat the routing, the health of servers or the per-client limits: it covers timeouts, long streams, session affinity and replay.
What you will learn
The lesson starts from a proxy whose bounds all have the same value and ends with a proxy where every duration has a meaning.
- Tell apart the four bounds
timeout connect,timeout queue,timeout serverandtimeout tunnel, and the phase each one watches. - Set different bounds so that the elapsed time points to the phase.
- Keep an SSE event stream and a WebSocket alive through silences.
- Choose a session policy and say what it does not guarantee.
- Prevent a lost response from making the same data be written twice, without depriving reads of their retries.
Prerequisites
The lesson assumes a running HAProxy 3.4 and a sudo account. The reference versions are those of the lab (HAProxy 3.4.7, Debian 13). The configuration excerpts of this page show the shape of the fix, not a configuration to copy as it stands: adapt the values yourself.
- Read an HAProxy configuration (
defaults,frontend,backend): Install HAProxy and read its configuration sections. - Route to several backends by ACL: Route by names and paths with ACLs and maps.
- Know what a health check is: Choose the load-balancing algorithm and measure health checks.
- Query the admin socket, because the lab takes a server out of service through this socket: Change HAProxy's state live with limited rights.
What does this lesson cover, and what does it leave to the neighboring lessons?
This lesson sets time, the lifetime of streams and replay; the neighboring lessons keep the rest. Per-client rate limits with stick-tables live in Limit abuse with HAProxy stick-tables. Load balancing, health checks and draining a server without cutting streams are in Choose the load-balancing algorithm and measure health checks, and the loss of an entry node in Test the availability of the HAProxy entry point. Reading the termination codes of the log is in Diagnose HAProxy without masking the cause.
Here, a session affinity (bringing a client back to the same instance) serves one purpose only: showing what it does not replace. The gRPC protocol, which needs other settings, is not covered. To see the same long streams on the Nginx side, read Nginx: publish WebSocket, SSE and gRPC without interruption.
Where do we start from, and what do we fix?
The lab starts from an API with two instances, a and b, behind an HAProxy with four things wrong. Both instances keep their sessions in their own memory. An SSE (Server-Sent Events, an HTTP stream where the server pushes events as they happen) stream and a WebSocket (an HTTP connection "switched" into a bidirectional channel, at the client's request through the Upgrade header) complete the application. The stream's heartbeat, that is, the interval between two events, is deliberately slow: 12 seconds.
The four faults: all the bounds equal 10 seconds and there is no queue bound; ten seconds of inactivity cut the stream and the WebSocket, for lack of a tunnel bound; retry-on all-retryable-errors replays everything that can be replayed; no session policy exists, so one request out of two changes instance. A recette.txt file gives the expected values (2, 10, 20 and 60 seconds) as acceptance criteria, without saying which directives to write.
How does one bound per phase tell you which phase is slow?
When each phase has a different bound, the duration of an error identifies the phase at fault, without reading a single log. A request goes through three slow phases in series. It first waits for a place in the backend's queue, bounded by timeout queue. It then establishes the TCP connection to the server, bounded by timeout connect. Finally it waits for the response, bounded by timeout server.
Look at the diagram from left to right: each box carries the directive that bounds the phase, and the tunnel exists only after an accepted Upgrade. If the three phases share the same value, an address that does not answer, a full queue and a slow response give roughly the same signal after the same time, around ten seconds. Without timeout queue, HAProxy applies to the queue the bound of timeout connect, which confuses a full queue with an unreachable server.
The lab's acceptance criteria therefore set timeout connect 2s, timeout queue 10s and timeout server 20s, with two attempts at connecting, that is about five seconds for an unreachable address, 10 s for a full queue and 20 s for a response that is too slow. These values are the exercise's, not recommendations for your traffic.
defaults mode http timeout connect 2s timeout queue 10s timeout server 20s timeout client 30s timeout tunnel 60s retries 1The block gathers the lab's bounds. With retries 1, HAProxy tries the connection once, then once more: two attempts. The total is nevertheless not twice the bound, because the manual adds between two tries on the same server a pause worth min(timeout connect, 1 s). Here that makes 2 s, then 1 s of waiting, then 2 s: five seconds and not four. It is the kind of gap you only find by timing, never by reading the configuration. The value of timeout client (30 s here), which bounds inactivity on the client side, is an example that the lab does not measure. To check, force each phase one after the other and time it, here for the connection:
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' --max-time 40 \ http://<proxy>/injection/connexionThe output shows the HTTP code then the duration in seconds, which should come close to five. The lab's test is content to require that the three durations be at least three seconds apart: it measures the distinction between the phases, not the exact value of each.
The HTTP code is the second half of the diagnosis. The lab's test requires a 504 when the response exceeds its bound, which says the server gave up on the response side. For the connection and the queue, HAProxy's documentation describes a 503; the lab does not measure it and keeps only the duration.
How do you keep an SSE stream or a WebSocket alive through silences?
timeout server measures an inactivity, not a total duration: an SSE stream survives as long as the silence between two events stays shorter than this bound, and a WebSocket falls under timeout tunnel. The bound restarts from zero at every byte received. With a 12-second heartbeat, a timeout server 10s cuts a stream that is nevertheless healthy; timeout server 20s lets it live as long as it beats.
The setting is therefore read against the application's heartbeat, not against the duration of the stream. Asking for a 30-second stream does not require a 30-second bound: it is enough that timeout server exceeds the longest silence the application produces. That is what avoids keeping open, for nothing, a connection whose server is dead.
The WebSocket is another case. After the 101 Switching Protocols response, the switched connection becomes a tunnel: HAProxy relays the bytes without interpreting HTTP any more. It is then timeout tunnel that decides the length of a tolerable silence, and no other directive replaces it. A chat window left open with nobody writing exchanges nothing for minutes: that silence is normal, and timeout tunnel 60s covers it beyond the test's 30 seconds.
To check, the test plays both streams against the proxy. The SSE stream must return several timestamped events, spaced by the heartbeat, over 30 seconds. The WebSocket returns its first message, stays silent for 30 seconds, then must still answer. If the last exchange is missing, it is the tunnel bound that closed the connection.
One last point: timeout tunnel does not lift all limits. It replaces neither timeout client, nor the bounds of an ordinary HTTP request, and a value that is too large keeps dead connections open. Set it to the length of silence you really accept.
How do you stick a session to an instance, and how far does it hold?
An affinity cookie set by HAProxy sends a client back to the instance that holds its session, but it does not copy the session: taking the instance out loses it. When sessions live in the memory of each instance, a balance roundrobin sends the next read to the other instance, which does not know the session. The lab shows it: without a policy, one read out of two fails.
The simplest form lives in the backend. HAProxy adds a cookie named SERVEUR there, whose value designates the chosen server, and each server carries the value that matches it.
backend applicatif balance roundrobin cookie SERVEUR insert indirect server a 127.0.0.1:18181 check cookie a server b 127.0.0.1:18182 check cookie bThe word insert makes HAProxy write the cookie, and indirect removes it before forwarding the request to the application, which therefore has nothing to know about it. The httponly keyword is deliberately absent: without it, the cookie does not carry the HttpOnly attribute (which forbids the page's JavaScript from reading the cookie) and a client script can read it. For a real browser, add httponly and re-read these attributes against your cookie policy. To check, open a session with curl -c biscuits then read it twenty times with curl -b biscuits: every read must return 200.
Another key exists, the source address (balance source). It also passes the test, but behind a corporate NAT all the clients share an address and a single instance receives everything. The cookie sticks a client, the address sticks a network.
The limit is then measured: the test takes the instance that holds the session out of service, with set server applicatif/<name> state maint on the socket, and reads again. The expected response is a 409, unknown session, because the other instance has nothing. This is not a defect to fix: sticking is not replicating. Every rolling update that empties an instance disconnects its ongoing sessions. To keep them, you have to move the state out of the instance, to a shared database or cache. That choice belongs to the application, not to the proxy.
How do you prevent a lost response from being written twice?
HAProxy must replay a request only if it never reached the server: retry-on conn-failure is enough for that, whereas all-retryable-errors also replays writes whose response was lost. A replay (retry) sends the same request again to a server after a failure. For a read, it is harmless. For a non-idempotent write, a request whose two executions do not produce the same state as one, it is a duplicate.
In the lab, the write endpoint /ecrire counts the write in its business counter, then closes the connection without answering. The work took place, but the client received nothing. With retry-on all-retryable-errors, HAProxy sees a connection closed without a response, believes in a failure and sends the request again: the counter goes to two. The client cannot know it: only the backend says how many times the work took place.
defaults retries 1 retry-on conn-failureconn-failure is the default value of retry-on: it replays only a connection that failed before reaching the server, therefore before any work. You can also state the rule by method with http-request disable-l7-retry if METH_POST, which forbids replaying a POST at the application level. The two gestures are not mutually exclusive.
The half that matters comes after. Restricting retries is not the same as removing them. In the bascule backend, one server is dead and another is healthy, with option redispatch. A read whose connection fails must still reach the server that answers; that is what the test checks over six reads. A configuration that removed all replay would gain safety by losing its fault tolerance.
To check the write, count before and after a single attempt, through the business counters of both instances: the difference must be exactly one.
What does the lab validate, and what does it leave to your judgment?
The lab validates measured behaviors, not directives or the substance of a text. That is why an honest page also says what it does not measure.
- It measures three durations played separately (connection, queue, response), at least three seconds apart and close to their bound, plus the 504 code for the slow response.
- It measures the SSE stream over 30 seconds, and the WebSocket that still answers after 30 seconds of silence.
- It measures twenty successful session reads, then the 409 once the instance is removed.
- It measures a write counter equal to one after one attempt, and six successful reads on the backend with a dead server.
- It does not measure the exact value of
timeout tunnelor oftimeout client, nor the error code of the queue or of the connection, nor which directive you use to stick the session. - It reads the decision record by keyword search, with a minimum size: it is a safeguard, not proof of understanding. Write it for an operator who will do a rolling update tomorrow.
Practice
The lab gives you the faulty proxy and asks you to fix it: distinct bounds for the connection, the queue and the response, holding the SSE stream and the WebSocket, a session cookie, then restricted retries without losing those of reads. You also write a short decision record that says what happens to a session when its instance leaves service. Validation covers the behavior: it times the three phases, replays the streams, reads the session twenty times before removing its instance, counts the writes of one attempt and reads the backend with a dead server. The writing of the decision record is only checked by keywords.
Key points
- Three slow phases, three different bounds: the elapsed time points to the phase, the connection (
timeout connect), the queue (timeout queue) or the response (timeout server). - Without
timeout queue, waiting in the queue falls back on the connection bound and is confused with an unreachable server. timeout serveris an inactivity bound: set it above the application's heartbeat, not the stream's duration.- A switched WebSocket falls under
timeout tunnel, which no other directive replaces. - An affinity cookie sticks a client to an instance; sticking is not replicating, and taking the instance out loses the session.
retry-on all-retryable-errorsreplays a write whose response was lost;conn-failurereplays only what did not reach the server.- Restricting retries is not removing them: a read must still reach the healthy server.
Knowledge check
These questions cover the phase that each bound watches and what a replay does to a write. Being able to quote the four directives is not enough: you must predict what the proxy will do before measuring it.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Change HAProxy's state live with limited rights
A new full quiz with random questions
Work again only on the questions you missed