test

Every observability rollout I have joined mid-flight had the same shape: hundreds of dashboards, thousands of metrics, a Grafana instance nobody could navigate — and no answer to the only question that matters: "Is the service okay right now?"

The team is proud of the coverage. Every microservice has a dashboard. Every dashboard has twenty panels. CPU, memory, GC pauses, connection pools, queue depths — all graphed, all colorful, all ignored. Then an incident hits, and the on-call engineer opens six dashboards, squints at forty squiggly lines, and asks in Slack: "does anyone know if this is bad?"

That question is the tell. If your team can graph everything but can't say whether users are having a good experience, you didn't build observability. You built decoration.

Dashboards Answer "What." SLOs Answer "So What."

A dashboard shows you that p99 latency is 840ms. Fine. Is that a problem? Should someone be paged? Should the release train stop? The dashboard has no opinion — and that's the flaw. Every panel you add without a defined target just increases the number of things that look concerning at 3 AM.

A Service Level Objective forces the conversation the dashboards let you skip:

What do users actually experience? Not what the pods are doing — what a person clicking "checkout" or calling your API sees.
How do we measure it? Which metric, measured where, is the honest proxy for that experience?
How much failure is acceptable? Because 100% is not a target, it's a fantasy — and pretending otherwise means you have no principled way to decide when to stop shipping features and start fixing reliability.

An SLO is not a metric. It's an agreement — between engineering, product, and reality — about what "good enough" means. The metric is just how you check whether you're keeping the agreement.

Why Teams Do It Backwards

Building dashboards first is seductive for reasons that have nothing to do with reliability. Dashboards are visible progress — you can demo them. They require no negotiation — nobody has to agree on a number. And the tooling nudges you there: every vendor onboarding flow starts with "import these 40 pre-built dashboards."

Setting an SLO, by contrast, is uncomfortable. You have to admit that some failure is acceptable, put a number on it, and defend that number to a product manager. It's a political act as much as a technical one. So teams defer it — and drown in panels instead.

Start Embarrassingly Small

The biggest SLO mistake after not having one is trying to have fifty. Here's the loop I set up on every engagement, and it's deliberately tiny:

Pick one user-facing journey. The one that pays the bills: checkout, login, the core API endpoint. Not an internal service — something a user touches.
Pick one metric. Usually request success rate or latency, measured at the load balancer or edge — the closest honest vantage point to the user. Resist the urge to instrument five golden signals on day one.
Pick one target you can defend. Look at the last 30 days of actual performance. If you've been delivering 99.92%, set the SLO at 99.9%. Don't declare 99.99% because it sounds professional — you'll blow the budget in week one and everyone will learn to ignore it.
Alert only on burn rate against that target. Delete — or at least mute — the CPU and memory alerts for that service. If users are fine, nobody should be woken up.

What a Burn-Rate Alert Looks Like

Burn rate measures how fast you're consuming your error budget. A multi-window setup catches both fast outages and slow bleeds. In Prometheus terms:

# Fast burn: paging alert
# 14.4x burn = exhausting a 30-day budget in ~2 days
- alert: SLOFastBurn
  expr: |
    (1 - success_rate:ratio_rate5m)  > (14.4 * 0.001)
    and
    (1 - success_rate:ratio_rate1h)  > (14.4 * 0.001)
  labels:
    severity: page

# Slow burn: ticket, not a page
- alert: SLOSlowBurn
  expr: |
    (1 - success_rate:ratio_rate6h)  > (3 * 0.001)
    and
    (1 - success_rate:ratio_rate3d)  > (3 * 0.001)
  labels:
    severity: ticket

Two alerts. That's the entire paging surface for the service. Compare that to the 30-alert rulebooks most teams carry, where half fire weekly and all get acknowledged without being read.

The Error Budget Is Where It Pays Off

Once the SLO exists, you get the error budget for free — and the error budget changes conversations that dashboards never could. When the budget is healthy, the team ships aggressively and nobody debates it. When the budget is burned, the conversation about pausing the release train stops being a personality contest between the loudest engineer and the most impatient PM. The number decides.

I've watched a single burn-rate chart end a recurring argument that had consumed months of standups. That's the return on investment: not prettier graphs, but decisions that make themselves.

Then — and Only Then — Build Dashboards

With the loop running (SLO → burn-rate alert → error budget conversation), you finally know what dashboards you need, because every panel now has a job: explaining why the SLO is burning. The structure writes itself:

Top level: SLO status and error budget remaining. One glance answers "are we okay?"
Second level: the request path — edge, load balancer, service, dependencies — to localize where the errors or latency come from.
Third level: resource detail (CPU, memory, connections), consulted only when the layers above point there.

In practice you'll find you need perhaps a tenth of the dashboards you were about to build — and the ones you keep will actually get looked at, because they sit on the path from "the SLO is burning" to "here's why."

The Takeaway

Observability isn't measured in panels. It's measured in the time between "something is wrong" and "we know what and how bad." SLOs are what give that question a shape. So before you import another dashboard pack: pick one journey, one metric, one defensible target. Alert on burn rate. Have the error budget conversation. Then build only the dashboards that serve that loop.

Everything else is decoration.

Running an observability rollout and not sure where to draw the SLO line? Reach out — this is exactly the kind of engagement I take on.