The difference between monitoring and observability in distributed systems

Classic monitoring is built around questions known in advance: whether the server is up, whether CPU is above 80 percent, whether the disk is filling, whether the nightly job finished. An operator picks a signal, sets a threshold and alerts when it is crossed. For a small number of servers running a monolith this works well, because the list of things that can go wrong is short and the people on call know it by heart.

The limitation is that every check encodes a prediction. When the failure is one nobody predicted, such as a slow downstream dependency, a retry storm or a cache that is technically up but returning stale data, the dashboards stay green while customers complain. Monitoring remains necessary, but it is no longer sufficient once the system is larger than one person’s mental model.

Move the same business function into a dozen services with a message queue in the middle and three things change. First, a single user request now touches many processes on many hosts, so no single log file tells the whole story. Second, failures become partial: one region, one tenant or one code path is slow while the aggregate looks healthy. Third, the system changes many times a day, so yesterday’s baseline no longer applies.

Observability is the property of being able to understand the internal state of the system from the data it emits, without shipping new code to ask a new question. In practice it means collecting rich, high-cardinality telemetry (which customer, which endpoint, which version, which region) and being able to slice it on demand. The question shifts from whether the system is up to which users experience slowness, since when, and what changed.

A service level objective is a target for a user-facing measure: for example, 99.9 percent of checkout requests complete successfully within 500 milliseconds, measured over 30 days. The percentage and the window are set by the organization; the point is that the SLO describes what users experience rather than what a host reports.

An SLO defines an error budget, the amount of failure that can be absorbed in the window, and with it a different alerting model. Instead of paging when CPU crosses a line, the team pages when the budget is being consumed too fast to last the window (a burn-rate alert). Slow burns become tickets for the next working day; fast burns page the on-call engineer. This single change removes most of the alerts that fire without anyone needing to act.

SLOs also give engineering and the business a shared language. If the budget is healthy, the team can ship faster; if it is being spent, the team invests in reliability. A CPU graph cannot support that conversation.

The most common failure of monitoring programs is an excess of alerts rather than a shortage. When a rotation receives dozens of pages a week and most require no action, people stop reading them, and the one that matters is lost in the noise. Every alert should meet a simple test: a person must be able to do something about it immediately. If that is not the case, it belongs on a dashboard panel or in a daily report rather than in a page.

Treat the alert set as code under review. Each alert has an owner, a runbook link and a reason to exist; alerts that fire without action for a month are deleted or demoted. Measure the rotation itself: pages per week, pages outside working hours, time to acknowledge. A healthy on-call rotation is a sign of a healthy system, and an unhealthy one is a clear signal of where to invest.

Distributed systems add one more requirement: alerts should carry context. A page that reads “error rate high on payments-api” is a research task; a page that links to the affected SLO, the latest deployment and the top failing endpoint is a decision.

Observability data has a cost, and the bill tends to arrive after the habit has formed. The two drivers are volume (bytes of logs and spans per request, multiplied by traffic) and cardinality (the number of distinct label combinations on a metric). A metric labeled with user ID or request ID produces a new time series for each value and overwhelms any metrics store.

Sensible defaults go a long way. Keep high-cardinality identifiers in traces and structured logs rather than in metric labels. Sample traces, keeping every error and slow request and a fraction of the normal ones. Set retention by signal: metrics for a year at reduced resolution, logs for weeks, traces for days, with an archive tier for anything compliance requires. Review the top producers of log volume monthly; a single verbose debug statement in a hot path can account for a large share of the bill.

The goal is to be able to answer the next unknown question quickly, at a cost that the organization is willing to keep paying, rather than to collect everything.