Observability is the ability to understand what happens inside a system by reading only what it produces: its logs, its metrics and its traces. Where monitoring detects the failures you planned for, observability lets you diagnose the ones nobody imagined. This lesson is for developers and ops engineers who run distributed systems: it explains the three pillars, how a shared identifier correlates them, the cardinality trap and the place of OpenTelemetry.
It is 3 a.m. A customer reports that their payments have been failing for an hour. Your dashboards show normal CPU, normal memory and a global error rate of 0.3%: everything is green. Yet something is wrong.
You spend two hours digging through the logs of six services, rebuilding the request's path like someone gluing pottery shards back together. At 5 a.m. you finally find it: a timeout on the banking API, triggered only above €500, because an external anti-fraud check was down. Your monitoring said all was well; your system simply was not observable.
What you will learn
This lesson extends SLOs and error budgets: an SLO can only be steered if you can measure the service and understand its deviations. It sets the vocabulary that the on-call lessons and incident lessons rely on.
- Distinguish monitoring, which answers planned questions, from observability, which explores the unknown.
- Identify the role and limits of the three pillars: logs, metrics, traces.
- Correlate signals through a
trace_idto diagnose in minutes rather than hours. - Control cardinality to avoid exploding storage costs.
- Choose a metrics framework, RED, USE or Golden Signals, and place OpenTelemetry.
- Understand why observability also serves security, and how it can leak secrets.
These goals fit a precise scope: the lesson keeps what a DevSecOps path needs, that is the three pillars, correlation, cardinality, the RED, USE and Golden Signals frameworks, OpenTelemetry and the link with security. The monitoring versus observability distinction is only sketched here (a table and two subsections). Its history, its detailed limits, the criteria for moving from one to the other and a full investigation scenario are in the lesson Observability vs monitoring (in French) of the Observability course.
What is the difference between monitoring and observability?
Monitoring checks thresholds you defined in advance; observability lets you ask, after the fact, questions you had not planned. The two terms are often treated as synonyms, yet they meet two distinct needs. The table sums up the gap, which the two subsections detail.
| Aspect | Monitoring | Observability |
|---|---|---|
| Question | "Does X exceed threshold Y?" | "Why is the system behaving this way?" |
| Approach | Predefined questions | Ad hoc questions, exploration |
| Strength | Detecting known problems | Diagnosing unknown problems |
| Limit | Only sees what it is asked | Needs rich instrumentation |
Monitoring answers known questions
Traditional monitoring rests on a simple principle: you decide in advance what to watch, you set thresholds, and an alert fires when one is crossed. It is an approach based on predefined questions.
A "CPU > 90%" alert asks a question framed upfront: "does the CPU exceed 90%?". Monitoring therefore excels at known failures, the ones you have already lived through or anticipated. Faced with the unexpected, it stays silent: when a user says "it's slow" and everything is green, it can only answer the questions it was taught.
Observability lets you ask new questions
A system is observable if you can understand its internal state by analysing only its outputs, without changing its code or redeploying it. The term comes from control theory, where it names exactly this property.
Back to the opening scenario. With an observable system, you start from the symptom ("failed payments for this customer"), filter that customer's traces, see which service is at fault, then read that service's logs. Instead of two hours of digging, a few minutes of structured investigation.
What are the three pillars of observability?
The three pillars of observability are logs, which record each event, metrics, which aggregate measurements over time, and traces, which follow a request from service to service. Each lights up the system from a different angle, and their combination gives a complete understanding. The diagram shows how one identifier links them.
| Pillar | Role | Strength | Weakness |
|---|---|---|---|
| Logs | Detailed record of events | Debugging, audit, compliance | Huge volume, storage cost |
| Metrics | Aggregated overview | Trends, alerts, dashboards | No individual detail |
| Traces | Path of a request | Bottlenecks, dependencies | Complex to set up |
Logs: the detailed record
A log is a timestamped record of an event: it tells what happened in the system, line after line, like a ship's log. A well structured log looks like this:
{ "timestamp": "2026-01-10T14:32:15.123Z", "level": "ERROR", "service": "payment-api", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "user_id": "usr_456", "message": "Payment failed", "error_code": "CARD_DECLINED", "amount": 99.99, "currency": "EUR"}Each field adds context. The timestamp says when, the service says where, and the trace_id links this log to the other events of the same request. The business fields (amount, currency) give the real impact. The trace_id here follows the W3C Trace Context standard format: 32 hexadecimal characters.
Logs shine for detailed debugging, audit and compliance: they prove who did what, and when. Their weakness is volume: an application can produce gigabytes a day, and answering "how many payment errors this week?" by scanning millions of lines remains tedious.
The guide Logs: understanding the richest signal (in French) details log levels, the collection pipeline and querying at scale.
Metrics: the overview
A metric is a numeric measurement tracked over time: how many requests, what response time, what error rate. Where logs tell individual stories, metrics give the overview. Here are two metrics in the exposition format of Prometheus, the most widespread metrics tool:
# HELP http_requests_total HTTP requests handled.# TYPE http_requests_total counterhttp_requests_total{service="payment-api",status="500"} 142# HELP http_request_duration_seconds HTTP request duration.# TYPE http_request_duration_seconds summaryhttp_request_duration_seconds{service="payment-api",quantile="0.99"} 2.3http_request_duration_seconds_sum{service="payment-api"} 5120.4http_request_duration_seconds_count{service="payment-api"} 12873The first counts requests that ended in an HTTP 500 error. The second gives the response time at the 99th percentile: 2.3 seconds means 99% of requests are served in under 2.3 seconds. The labels in braces (service, status) split each metric into time series.
Metrics come in four families. Counters only go up (total requests, cumulative errors). Gauges go up and down (memory used, active connections). Histograms distribute values into intervals (buckets) to rebuild a distribution at query time. Summaries compute percentiles on the application side, like the quantile="0.99" above: lighter to query, but impossible to aggregate across several instances of the same service.
Naming can be checked with promtool, the tool shipped with Prometheus. On the example above saved as metrics.txt, the following command prints nothing and exits with code 0 (tested with Prometheus 3.7.3):
docker run --rm -i --entrypoint promtool \ prom/prometheus:v3.7.3@sha256:49214755b6153f90a597adcbff0252cc61069f8ab69ce8411285cd4a560e8038 \ check metrics < metrics.txtA metric named api_latency_ms, on the other hand, is rejected with code 3 and this message: metric names should not contain abbreviated units. The Prometheus convention requires base units spelled out, hence http_request_duration_seconds.
Metrics shine for trends and alerts: "is the error rate rising?" is a trivial question with them, tedious with logs alone. But they do not tell the why, and they lose individual detail: you cannot find in them the exact request that failed for one customer. The guide Metrics: the compact and fast signal (in French) details the naming conventions of each type.
Traces: the path of the request
A trace follows the path of a request through the services of a distributed system. In a microservices architecture, a request can cross ten services before returning to the user; the trace captures that journey. It is made of spans, one span per operation: an HTTP call, a database query, an internal step.
Trace: 4bf92f35… (total duration: 850 ms)├── api-gateway (12 ms)│ └── auth-service (45 ms)├── payment-api (650 ms) ← bottleneck│ ├── fraud-check (120 ms)│ └── bank-api (480 ms) ← slow external call└── notification-service (35 ms)At a glance, you see that 650 ms out of 850 ms are spent in payment-api, 480 ms of them in the external call to the banking API. The bottleneck jumps out, and the dependencies between services appear, including calls the team did not know existed.
The trade-off is setup: every service must propagate the trace context to the next one. At scale, sampling becomes necessary, because you do not store 100% of traces. The guide Distributed traces: spans, propagation and sampling (in French) details W3C propagation and sampling strategies.
How do you correlate logs, metrics and traces?
You correlate the three pillars with a unique identifier, the trace_id, created when each request enters the system, then propagated to every service and written into every log and every span. On its own, each pillar stays partial: metrics say what, traces say where, logs say why. Linked, they form a continuous diagnostic path.
In practice, an alert fires: "HTTP 500 error rate > 5% on payment-api". The metric warns you. You filter the failed traces of the last fifteen minutes and see that 80% of them have a bank-api span in timeout: the trace shows where.
You open one of these traces and read the associated logs through their trace_id: "External API returned RATE_LIMIT_EXCEEDED". The log explains why. Within five minutes, the cause is found: a reconciliation batch launched by finance saturated the banking API's quotas. You suspend the batch, and the errors stop.
Without correlation, you would have had three tools and three partial views, to link by hand. The same thread then feeds the postmortem, which the lesson Incidents and postmortems teaches you to run. The guide Signal correlation (in French) details trace_id propagation, including across message queues.
Why does cardinality make costs explode?
The cardinality of a metric is the number of unique combinations of its label values; each combination creates a time series to store and query. A metric with a service label (5 values) and a status label (5 values) produces 25 series: low cardinality, perfectly manageable.
Add a user_id label across one million users, and you move to 5 × 5 × 1,000,000, that is 25 million series. The metrics database grows accordingly, and queries that answered in milliseconds become slow or fail. This is the invisible trap: one line of code accepted in review is enough to trigger it.
The rule is simple: never a label with unbounded values. User, request or session identifiers, email addresses: these values go into logs, designed to store individual events, and not into metrics, designed to aggregate. Labels with controlled cardinality have a finite, predictable number of values: service, endpoint, status_code, region, environment.
Which metrics should you track first: RED, USE or Golden Signals?
To get started, track the Golden Signals on your critical services, RED on every service that receives requests, and USE on resources (CPU, memory, disk, network). These three frameworks spare you from starting from scratch on a new system, and they overlap widely. The table helps you choose.
| Framework | Target | Metrics | When to use it |
|---|---|---|---|
| RED | Services (APIs) | Rate, Errors, Duration | Microservices, user-facing endpoints |
| USE | Resources (infra) | Utilization, Saturation, Errors | CPU, memory, disk, network |
| Golden Signals | Critical services | Latency, Traffic, Errors, Saturation | Production alerting, SRE |
The RED method for services
The RED method, proposed by Tom Wilkie, measures three things on any service that receives requests. Rate: how many requests per second. Errors: what share fails. Duration: how long they take. A drop in throughput can betray an upstream failure, a rise in errors a regression, a rise in duration an approaching saturation.
The USE method for resources
The USE method, formalised by Brendan Gregg, applies to every resource. Utilization: what share of capacity is busy. Saturation: is work waiting its turn? Errors: does the resource return errors? A CPU at 75% with an empty run queue is healthy; the same CPU with a growing queue is saturated, even if its utilisation does not look critical.
Google SRE's Golden Signals
The four golden signals come from Google's Site Reliability Engineering book, in the chapter on monitoring distributed systems. Latency measures response time, separating successes from errors: a fast error does not mean the same as an error after a timeout. Traffic quantifies demand, Errors count failures, and Saturation shows the remaining capacity headroom. The book presents them as the minimum to track if you can only measure four metrics.
What does OpenTelemetry bring?
OpenTelemetry (OTel) is the open standard that lets you instrument an application only once for its logs, metrics and traces, then send that data to any compatible tool. Before it, each tool imposed its own formats and SDKs: moving from Jaeger to Zipkin, or from Prometheus to Datadog, meant re-instrumenting all the code.
The project is hosted by the CNCF (Cloud Native Computing Foundation), which announced its graduation on 21 May 2026. That is its highest maturity level, the one Kubernetes and Prometheus hold. Its exchange protocol, OTLP, has become the common language of agents and backends.
The architecture has three parts. The application is instrumented with the OTel SDK. The Collector receives data from several applications, filters or samples it, then routes it to one or more destinations. The backends store and analyse. This decoupling lets you change backend without touching application code.
A fourth signal is progressing: continuous profiling, CPU and memory captures at code level, correlated with traces. On the OpenTelemetry side, the profiles signal has only been in public alpha since March 2026: the three historical pillars remain the foundation to master first. The guide OpenTelemetry: the universal instrumentation standard (in French) walks through the API, the SDK, the Collector and OTLP step by step.
Why does observability also serve security?
Observability serves security because the same signals that explain an outage also reveal an attack: a spike in authentication errors, an outbound call to an unknown domain, a service suddenly reading a database it never queries. Without this data, an intrusion looks like a performance incident, or goes unnoticed.
Logs provide the audit trail: who logged in, who changed what, from where. Traces draw the real map of calls, and a call that was not there yesterday deserves a question. Runtime detection, such as the one in the lesson Runtime detection with Falco, relies on these same events to raise a security alert.
The flip side is real: observability is also a leak source. A log that writes an Authorization header, a trace that captures a token in its attributes, a label that carries an email address: this data flows to tools far more people can read than the production database. The guide Secret leaks: logs, Docker, variables (in French) shows how to mask these values before they leave the application.
Which good practices should you adopt day to day?
Observability is not a tool you install once, but a discipline built into the development of every service. The practices below are grouped by pillar; they avoid the traps described above and keep costs under control.
For logs, adopt a structured format (JSON) with a shared schema from day one, and always include the trace_id. Keep the ERROR level for real errors, sample DEBUG logs in production, and filter sensitive data before writing.
For metrics, treat cardinality as a golden rule. Prefer percentiles (p50, p90, p99) to the average, which hides the extreme values users actually suffer. Name metrics in base units, checked by promtool, and alert on symptoms (latency seen by the user) rather than on causes (high CPU).
For traces, propagate the context across all services, message queues included. Sample with care: every failed request, a fraction of normal traffic. Add business attributes to spans and instrument external calls, where latency often hides.
Finally, an alert that rings for nothing and an investigation redone by hand every week are toil, the repetitive work that the lesson Eliminating toil teaches you to reduce. To go beyond this lesson, the section Observability fundamentals (in French) covers each pillar and cost governance in detail.
Key points
- Monitoring answers the questions you anticipated; observability lets you ask new ones when facing the unknown.
- The three pillars each bring an angle: logs the detail, metrics the trend, traces the path.
- Correlation through a shared
trace_idturns three partial views into a diagnosis of a few minutes. - Cardinality is the cost factor to watch: unbounded values (
user_id,request_id) go into logs, never into metric labels. - RED, USE and Golden Signals frame your metrics depending on whether you watch a service, a resource or a critical system.
- OpenTelemetry, graduated by the CNCF on 21 May 2026, lets you instrument once and change backend without touching code.
- Observability serves security (audit, detection) and can also leak secrets: filter what goes into logs and traces.
Knowledge check
Check that you have the essentials of this guide. The questions only cover what has just been explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
Continue the course: SLOs, SLIs and error budgets: measuring and steering reliability
A new full quiz with random questions
Work again only on the questions you missed
FAQ: frequently asked questions
user_id can multiply time series by millions and make the storage cost explode. The rule is simple: unbounded values go into logs, never into metric labels.Authorization header or a trace that captures a token sends that secret to tools many people can read. Mask these values before they leave the application.