Deploying a cloud infrastructure takes a few hours; operating it lasts years, and that is where the bad surprises happen. The provider takes over part of the operations work, but it also decides, on your behalf, when your machine restarts and on what date your database changes version. This lesson describes what stays your job day to day, the imposed events you must anticipate, and the operating traps specific to the cloud: quotas, the cost of observability (the ability to understand the state of a system through its logs, metrics and traces) and capacity, which is measured in account limits rather than in hardware.
What you will learn
- Tell what the provider operates from what stays with you.
- Anticipate the restarts and version upgrades it imposes on you.
- Recognise a quota that has been reached, which looks like an application outage, and name the families of limits and their traps.
- Control the cost of observability by acting on retention and emission.
- Plan capacity by projecting consumption onto the account limits.
Prerequisite: having understood the split of responsibilities between provider and customer, presented in Shared responsibility.
This lesson does not redefine the SLIs and SLOs (service level indicators and objectives), nor the principles of a good alert: those subjects are covered in depth in SLI, SLO and error budget (in French) and Effective alerting (in French), and they do not change because you are in the cloud. What changes is what follows. Logs, alerts, incident handling and changes have their own lesson, Operating: logs, alerts, incidents and changes (in French).
Who operates what, exactly?
The provider operates the infrastructure under your service; you operate everything above it. The line moves with the type of service, and it is precisely this moving line that produces misunderstandings, as the page on shared responsibility explains. The table compares three service levels: a virtual machine that you administer, a managed database (the provider operates it, you use it) and a serverless function (code run on demand, with no server to manage).
| Operations task | Virtual machine | Managed database | Serverless function |
|---|---|---|---|
| Replace a failed disk | provider | provider | provider |
| Apply operating system patches | you | provider | provider |
| Decide the engine version | you | you, within an imposed window | provider |
| Watch disk saturation | you | you | not applicable |
| Back up and verify the restore | you | you | you |
| Answer an alert at 3 a.m. | you | you | you |
The last line is the most important and the one most often forgotten. No service model, not even serverless, transfers the responsibility for your application availability to the provider. It guarantees its service, not yours: if your code calls a saturated database, nobody but your team will get out of bed.
The events the provider imposes on you
Your virtual machine will restart one day without you having decided it. The physical hardware that hosts it must be updated or replaced, and the three major providers handle this moment differently. It is a selection criterion that is rarely looked at, yet it shapes your operations work.
| Provider | Behaviour during host maintenance | What you must plan |
|---|---|---|
| Google Cloud | live migration of the machine, with no interruption for most types | little preparation, except for the types excluded from migration |
| AWS | notification of a scheduled event, restart or scheduled retirement | act before the date, or suffer the restart |
| Azure | planned maintenance announced, metadata service you can query | trigger the switch yourself during a window you choose |
A retirement notification is not a monitoring alert, and that is the trap. It arrives by email at the account address, or in a health dashboard that nobody looks at. Treat this channel as an alert source in its own right: redirect it to the technical team, not to the administrative mailbox where it will end up unread.
Why a managed version upgrade needs preparing
A managed service does not spare you version upgrades, it only takes the calendar away from you. Each version of a database engine has an end-of-support date at the provider. Past that date, the provider decides: either you migrated before, or it imposes paid extended support (it keeps fixing the version, for an extra fee) or a migration. At Amazon RDS, for example, a MySQL or PostgreSQL database goes automatically into paid extended support after the end of standard support, then its version is upgraded by force at the end of that support. The extra cost is a good way to discover that you had forgotten the subject.
The operating rule that follows comes down to three gestures:
-
Inventory the engine versions you use, with the end-of-support date announced by the provider. Keep this inventory in the same place as everything else, in your infrastructure as code tool.
-
Test the upgrade on a restored copy, not on the staging environment that has drifted over the last six months. Restoring a production backup into an isolated account costs little money and little time, far less than a failed migration.
-
Trigger the migration yourself, in a window you control. Suffering the automatic migration means discovering an application incompatibility at 4 a.m. with nobody to fix it.
Quotas: the outage that does not look like an outage
A quota that has been reached does not show up as a refused invoice but as a resource creation that fails. It is a failure mode specific to the cloud, and a particularly confusing one: your code is correct, your permissions are correct, and yet the call fails.
The situations where this happens are always the same, and always at the worst moment:
- A blocked autoscaling because the region's compute quota (in vCPUs or in number of instances, depending on the provider) has been reached. The elasticity mechanism stops dead, with nothing broken in the usual sense.
- A failover that fails because the fallback region never had the quotas of the primary region. Quotas are set per region, and a region where you create nothing keeps the default values.
- A blocked deployment because of a quota on public IP addresses or security rules, never looked at because it seemed absurdly high.
This lesson deliberately reproduces no table of quota values. These limits change without notice, differ from one region to another and depend on the age and verification of the account: a table copied here would be wrong in six months, with no way for the reader to notice. What stays true are the families of limits and the reflex that goes with them. A quota limits a resource; a rate limit limits a number of API calls per second. Both exist to prevent abuse, and both are met in operations.
| Limit family | What it limits | Classic trap |
|---|---|---|
| Compute capacity | depending on the provider, a number of vCPUs (virtual processor cores) per region and per instance family, as at AWS, or a number of instances; the unit is read in the console | sizing in "number of servers" when the quota counts vCPUs |
| Public IP addresses | reserved addresses, including detached ones | a forgotten address uses up the quota and the bill |
| API throughput per object | a number of requests per second, often per prefix of a storage | putting all the objects under the same prefix |
| Function concurrency | simultaneous executions across the account | a runaway function exhausts the quota of the others |
| Backup objects | the number of snapshots (instant copies of a disk) or manual backups | snapshots never purged, sometimes against a hard limit |
The costliest trap is the unit error. Where the compute quota is counted in vCPUs, as at AWS, it does not count machines: with 2-vCPU instances, the same limit allows half as many machines as with 1-vCPU instances. A capacity planned in number of servers therefore hits a wall whose error message does not mention servers.
The second trap is the moment of discovery. A deployment plan does not consult the quotas: it is the execution that fails, midway, with part of the infrastructure already created and billed.
Four reflexes settle most of it, and they do not go stale:
- Check your real quotas in the provider console (at AWS, the Service Quotas service), for your account and your region, not in a table read elsewhere.
- Monitor quotas as metrics, with an alert at three quarters of the limit, rather than discovering the ceiling on the day of the peak.
- Request increases in advance. Some are self-service (you get them yourself in the console) and immediate, others are subject to support approval, with a delay counted in hours or days, which is incompatible with a recovery plan; a few limits are hard and cannot be raised.
- Check the failover region: a recovery plan assumes sufficient quotas over there too, which is never automatic. This is the oversight that turns a recovery exercise into an incident.
The scaling limits themselves, such as the start-up time of an instance or the ceiling of a managed service, are covered in elasticity and scalability.
How much does your observability cost?
In the cloud, monitoring is a service billed by usage, and the item that explodes is almost always log ingestion (sending logs and storing them at the provider). Metrics cost little, traces cost a moderate amount, logs are billed per gigabyte ingested and then per gigabyte stored. A chatty application put in debug mode on a Friday evening produces a memorable bill.
Three trade-offs are enough to keep this item under control:
- Separate the retention period (how long logs are kept) according to use. Diagnostic logs serve for a few days; audit logs must survive for months for regulatory reasons. Applying the same retention to both means paying the price of audit on the whole volume.
- Filter at emission, not at reading. A log you do not send costs nothing. A log sent and then excluded from a search has already been billed at ingestion.
- Sample traces. The principle is stable: keeping a fraction of nominal traces is enough to characterise a behaviour, and keeping all the traces in error gives the diagnosis. The sampling rate itself has no universal value. It depends on your request volume, your error rate and the chosen strategy: head-based sampling decides before knowing the outcome of the request, tail-based sampling decides after and can therefore keep slow or failed cases, and adaptive sampling adjusts the rate to the traffic. Start from a low fraction on a high-traffic service, and raise it on a lightly used service where too thin a sample would show nothing.
This item belongs to cost governance as much as to operations. It rarely appears in invoice reviews because it is spread over small lines, yet it often exceeds the cost of the compute it observes. The method to make it visible is the one from the page governance and costs (in French): tag, then measure per project.
The question of backups, which live by default in the account they protect, has its own lesson: Back up outside the account and prove the restore (in French).
Planning capacity when you no longer buy hardware
In the cloud, capacity planning is no longer about the hardware to order, but about the limits you will hit. The provider has the capacity you no longer need to buy, but your account is capped, and that cap is what arrives first.
Four quantities deserve a projection, because they evolve slowly and then block suddenly:
- Your account quotas, covered above: raising them takes time, sometimes counted in days.
- The address ranges of your networks, whose exhaustion prevents creating a subnet at the worst moment. The subject is covered in create and size a network (in French).
- The real availability of a machine type (a machine size: number of vCPUs and memory) in your region, in particular for machines with an accelerator, which can run short without any announcement.
- Reservation commitments reaching their end, whose expiry makes the bill jump all at once if nobody anticipated it.
The method comes down to one habit: a quarterly review that projects the consumption of the last twelve months onto the next twelve, and compares the result to the limits in force. It is little work, and it turns a surprise outage into an increase request filed three months earlier.
Replacing machines rather than fixing them, detecting drift, renewing certificates and absorbing a provider outage make up the other half of everyday operations. These four subjects are covered in the lesson Replace, detect drift, renew certificates, absorb the provider outage (in French).
Key points
- No service model transfers your application availability to the provider. It guarantees its service, not yours.
- Your machine will restart one day without your consent. Google Cloud live-migrates most machine types, AWS and Azure notify and let you act.
- Maintenance notifications arrive by email, not in your monitoring. This channel must be routed to the technical team.
- A managed service does not remove version upgrades, it takes the calendar away from you. Migrate yourself before the imposed date.
- A quota that has been reached looks like an application outage, never like a billing problem. It is counted in the provider's unit (often vCPUs, not machines), monitored like a metric, and raised in advance, failover region included.
- Log ingestion often exceeds the cost of the compute it observes. Filtering at emission and differentiating retentions settle most of it.
- Capacity is projected onto the account limits, not onto hardware: quotas, address ranges, machine type availability and reservation expiry dates, reviewed every quarter.
Knowledge check
Check that you have grasped the essentials of this guide. The questions cover only what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: Pépin: find what actually weakens your cloud
A new full quiz with random questions
Work again only on the questions you missed