Before you open a service to the public, you accept it, you prove you can lose it and get it back, and you know how to undo a cutover that goes wrong. This lesson is for someone who has already published a service behind an HTTPS entry point and has checked only one thing: that it starts. It shows how to build an acceptance test from a real client, restore a backup nobody has ever tried, undo a cutover cache included, and name what a lab does not demonstrate. It gives no tool-specific directive: the reasoning holds for any proxy.
What you will learn
The lesson goes from accepting a publication to admitting its limits.
- Build an acceptance matrix that covers the name, the certificate, the business route and a denial.
- Run that acceptance test from a client that is independent of the machine hosting the service.
- Prove that a backup restores, by replaying the loss of the configuration and of the state.
- Undo a failed cutover without waiting for a cache to expire.
- Tell apart what a rollback recovers from what it leaves as it is.
- Name what a lab acceptance test leaves to check before a real go-live.
Prerequisites
The lesson assumes that you can read an HTTP response, run a curl, and that you have already seen a published service behind an entry point.
- Read an HTTP exchange, its status codes and its headers: Read HTTP before configuring a proxy.
- Place the entry point on the path of a request: Publish a web service.
- Know the role of a cache and of a lifetime (TTL): Master timeouts, retries and caching without altering responses.
- The lab provides the machines and the service: you install nothing.
What does this lesson cover, and what does it leave to other pages?
This lesson covers the decision to go live: what you check before opening, how you go back, and what that return does not recover. A go-live is the moment when a publication stops being a team experiment and becomes a service that others depend on. An acceptance test is the set of checks that lets you say "accepted" or "rejected" before that moment.
Two neighbouring topics already exist on the site, and this lesson does not repeat them. The mechanics of a reload and of a cutover between two generations are read on the tool side: Reload, update and recover HAProxy for HAProxy, Update, restore and reboot Nginx for Nginx. The backup and the recovery plan of a whole system, with the 3-2-1 rule, are in Keep: backup, restore and recovery plan (DRP) (in French). Here, the angle is narrower: a single publication, and the gesture of accepting it, restoring it and undoing it.
Why accept a publication from an independent client?
You must accept from a client that is not the service's own machine, because the operator sees the service from the inside, whereas a real client goes through all of it. From the machine that hosts it, a curl on localhost succeeds even if the name is not declared in the DNS (the service that turns a name into an address), even if the certificate does not match the name, even if a firewall cuts off access from outside. An independent client is another machine, with its own resolver and its own list of trusted authorities: it is the only position that tests the whole chain.
The lab scenario takes this case: a service is already in place, a business route behind TLS (the encryption that authenticates the server), served by two origins, one active and one candidate for a future cutover. The only check already done is that it starts: that says the process is running, not that a client gets the right response.
We recommend designating this acceptance client from the start and never confusing it with the operator. In the lab, it is the poste.lab machine. The proof of reachability rests on what it observes there, not on what the operator sees from edge.lab.
What must an acceptance matrix cover?
An acceptance matrix crosses what you expect from a publication with the measurement that demonstrates it, and it covers at least the name, the certificate, the business route and a denial. The table below reads line by line: you start from the expectation, look at the measurement made from the client, then at the pitfall it avoids.
| Expectation | Measurement from the client | Pitfall avoided |
|---|---|---|
| The name leads to the service | dig service.mes.test A queries the client's resolver | A workaround with --resolve or /etc/hosts hides a missing DNS entry |
| The certificate is trusted | curl --cacert ca.pem https://… validates the chain | -k (--insecure) accepts any certificate |
| The business route answers | Status 200 and a body that contains the expected field | A 200 from a custom error page proves nothing |
| An internal route is denied | Status 403 on the management route | An exposed internal route goes unnoticed until you ask for it |
A certificate authority (CA) is the body whose signature attests that a certificate really belongs to a name. In the lab, the CA is a test authority whose certificate is placed on the client: the TLS check designates it explicitly with --cacert, and never uses --insecure. Keep that habit outside the lab too: a check that disables verification no longer checks the certificate.
The last line deserves a pause. A denial is an expected negative response: the service must answer 403 to whoever asks for a management route. An acceptance test that checks only what must work lets through what should not work. In the lab, the starting configuration exposes this internal route by mistake; the acceptance test finds it, and the fix goes through the configuration, not through the acceptance test.
How do you write an acceptance test that tells the truth?
An honest acceptance test declares its results after measuring them, and it must be able to fail: a grid that answers "accepted" in every case checks nothing. The lab asks for an acceptance file that declares, for DNS, TLS, HTTP and the denial, that the measurement passes. The test does not just read these declarations: it replays the measurements itself from the client.
Here is the working order we recommend, and it is a habit of caution rather than a protocol rule. First, run each measurement and read its output. Then fix what fails at the source, in the service configuration. Finally, rewrite the grid with the results obtained after the fix, without touching a cell to make it pass.
The check must go through the client's resolver, that is, the server the machine asks for addresses. If you force the address with curl --resolve, the request succeeds, but you have learned nothing about the DNS: the lab checks it separately with dig. To prepare the name and its cutover, see Prepare DNS names and their cutover.
Why is a backup that was never restored only a belief?
A backup is proven only the day it has been restored: until then, you believe it works, you do not know it. The scenario says so plainly: a backup whose content nobody has ever restored is a belief, not a fact. The failures that an exercise reveals are ordinary: an empty file, a path that has changed, a missing access right.
Restoring a publication involves two distinct elements. The configuration describes how the service routes and protects itself. The state is the data that the service produces and that you cannot recreate, here a stock level. Losing one without the other gives a service that starts with wrong values: a configuration restored onto a default state answers 200, but with a stock that matches nothing.
That is why the lab measures the restored value and not only the status. After the simulated loss, the business route must no longer answer 200, which proves the loss was real. Once the procedure has run, it must answer 200 within fifteen seconds, with the original stock, not a default value. A service that answers 200 with an invented figure is, in our view, worse than a stopped service: nobody notices.
How do you prepare the restore before the loss?
We recommend writing and trying the restore procedure before anything has been lost, because a procedure written during the incident is written under pressure, by someone who has never restored. The lab therefore requires a restore script to exist before the controlled loss. The test checks it like this: the file must exist, then the test deletes the active configuration and state, then it runs the script.
A useful restore procedure answers four questions. Where is the backup, and is it readable from the machine that must restore it? What must be put back, configuration and state together? How does the service pick up what has just been restored, reload or restart? How do you check that the recovery is real, and not merely that the script finished without error? The last question matters: a script that exits with code 0 proves it met nothing abnormal, not that the service serves the right data.
We also recommend not leaving the backup copy on the machine it protects. In the lab, it sits right next to the service to simplify the exercise, which is not a model: a backup that disappears with the machine backs up nothing. The 3-2-1 rule (three copies, two media, one off-site) is described in the lesson on keeping data; it is common advice, not an obligation.
How do you undo a failed cutover, cache included?
Undoing a failed cutover means putting the old target back in service and purging the cache: changing the configuration alone leaves the cache returning the failing response until its lifetime ends. A cutover is the change of traffic target, here from the active origin to the candidate origin. A cache keeps a response to return it to later requests without querying the origin; its lifetime (TTL) says how long it keeps it.
The diagram contrasts the two ways of going back. On the first branch, you fix only the configuration: the cache has no reason to know the target has changed, and it still returns the failing response until its lifetime expires. On the second, you fix the configuration and then purge the cache: the client gets the right response as soon as the purge is done. Look at the time between the error and the recovery: that is what the lab measures.
In the lab, the cache lifetime is set to 24 seconds in the configuration. The cutover injected towards the candidate origin, which is invalid, makes the business route answer 500. The test then runs your rollback script and requires the route to answer 200 in under six seconds: simply waiting for the 24 seconds of the lifetime would not pass. The test thus judges that the return came from a decision, not from waiting. That is the point of the requirement: a rollback that works only if you wait is not a procedure, it is luck.
What does a rollback not undo?
A rollback puts the configuration back in its previous state, but it undoes neither the effects already produced nor the copies of the response that live elsewhere. Three families of effects escape it, and you need to know them before promising "we roll back in one command".
The first is the business state written during the cutover, the service's data. If the failing target accepted writes before being fixed, going back to the old target neither recreates nor removes those writes. A restore of the state can undo them, at the price of losing what has been written since the backup. The configuration rollback and the data restore are therefore two distinct gestures, with distinct costs.
The second is the copy that a third party has kept, whose lifetime is beyond your reach. A browser, a DNS resolver or an intermediate cache that you do not control may have memorised a response or an address. You purge your cache, not theirs: the fix spreads according to their lifetime, not according to your decision. This is why the DNS is prepared before a cutover, by lowering the lifetime upstream.
The third is the generation you believe is retired, the one from the previous release. A release does not retire the generation it replaces, because old connections still talk to it; it retires the one that had been replaced at the previous release. This rule, detailed in Reload, update and recover HAProxy, implies that a realistic rollback can count on the previous generation, still present. We therefore recommend not parting with it too early.
One last reminder, often confused with the above: "no downtime on reload" does not mean "no downtime on a host failure". A clean reload preserves the connections of a process that keeps running. It does not protect against a machine that stops; that case belongs to recovery, hence to the restore, not to the rollback.
What does a lab acceptance test prove, and what does it leave out?
A lab acceptance test proves that the method works on a simulated bench, not that the real go-live will be compliant: it must end with a list of what it has not demonstrated. The lab ends with a report of limits, and that is deliberate. The scenario asks for an honest report of what the acceptance test still owes to a real go-live exposed on the Internet.
Four limits are named by the test, and each has a precise content. The public resolver: the lab name, in .test, is resolved by a private resolver; this suffix is reserved for testing (RFC 6761) and does not exist in the public DNS, so propagation, operators' caches and the presence of IPv4 and IPv6 addresses have not been tested. The external client: the acceptance test runs from a lab machine, not from the Internet; the edge firewall, network address translation (NAT) and the network paths of a real visitor are out of scope. The public authority: the certificate is signed by a test authority that the client received separately; a public browser does not trust it, and automatic renewal with a real authority has not been played. Finally the simulation: the loss was simulated by deleting files, not by losing a disk or a machine.
The test checks that these four elements are mentioned in the report, a file of a certain minimum length. To write it usefully, tie each limit to the check you will need to make on go-live day, for example "query the name from a public resolver" or "test from an outside network".
Practice
The lab entrusts you with a publication in service whose acceptance test, backup and rollback nobody has checked. You accept it from the independent client and fix what the acceptance test reveals. You then write a restore procedure and a rollback procedure, then the report of limits.
The validation runs the measurements itself: it queries the DNS from the client, checks the certificate with the lab's authority, requires 200 and 403, deletes the configuration and the state before running your restore, and compares the restored stock with the original value. It also injects a failed cutover and requires your rollback to return 200 in under six seconds.
Key points
- Accepting a publication is done from an independent client: DNS through the real resolver, TLS with the designated trusted authority and never
--insecure, business route and denial of an internal route. - An acceptance test that checks only what must work lets through what should not work: the denial is part of the matrix.
- A backup is proven only the day it is restored, configuration and state together, and the proof is about the restored value, not the status alone.
- We recommend writing and trying the restore procedure before the loss, not during it.
- Rolling back a cutover means putting the old target back and purging the cache: changing the configuration alone leaves the cache returning the failing response until the end of its lifetime.
- A rollback undoes neither the writes made during the cutover nor the copies kept by third parties, and a release does not retire the generation it replaces.
- "No downtime on reload" does not mean "no downtime on a host failure".
- A lab acceptance test ends with what it has not proven: public resolver, external client, public authority, simulation.
Knowledge check
These questions check that you can choose the measurement of an acceptance test, prove a restore and decide on a rollback.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Diagnose a web service layer by layer
A new full quiz with random questions
Work again only on the questions you missed