TL;DR. An availability SLO across thirteen Kubernetes clusters came down to one PromQL ratio, three decisions about what counts, and one blind spot I wrote down and shipped anyway. The decisions: health probes out of the denominator, client errors out of the error budget, and a window that opens on the calendar month. The first alone moved the number materially; the other two change what the number means.
The blind spot was worse than it looked. A request-based SLI scores the requests a service received, so a service that is completely down receives nothing, produces no errors, and drops out of the number without lowering it. Fixing that meant changing the denominator from requests to wall-clock time. The fix reintroduced the same class of bug one level up. None of this is in the query. All of it is a decision about what does not count.
One metric family, thirteen clusters, two gateways
The fleet is heterogeneous in the way fleets get when they grow one customer at a time. Some clusters front their traffic with Envoy Gateway, older ones with ingress-nginx. Some applications ship latency histograms, most ship only counters. Waiting for the fleet to become uniform before measuring it was not an option, so the SLO had to be built on whatever every cluster already exported.
That turned out to be one metric family: http_server_requests_seconds, the Micrometer counter every Spring Boot service emits. It carries cluster, application, uri, status and outcome labels, and it exists on all thirteen clusters regardless of which gateway sits in front. The metrics store is VictoriaMetrics, queried with PromQL from Grafana. The availability SLI is the obvious ratio, evaluated over whatever time range the dashboard is showing, which Grafana exposes as $__range:
100 * (
1 -
sum(increase(http_server_requests_seconds_count{outcome="SERVER_ERROR"}[$__range]))
/
sum(increase(http_server_requests_seconds_count[$__range]))
)
That query is not the design. The design is everything that had to be removed from it before the number meant anything. The dashboard, which I will call the request-based board from here on, is generated by a Python script that writes the Grafana JSON. That script is the generator, and its docstring is where every decision below was recorded, because an unstated exclusion is a lie.
Decision one: a health probe is not a request
Kubernetes probes every pod’s health endpoint several times a minute. Load balancers probe it again. On this fleet one application’s /management/health endpoint alone received about 6.7 million requests in 28 days, and it never fails, because failing would get the pod restarted.
Left in the denominator, those requests dilute every real error. An outage on the customer-facing endpoints shrinks to a rounding error under a pile of successful probes. So probe URIs come out of both sides of the ratio:
http_server_requests_seconds_count{uri!~"/actuator.*|/management.*"}
A side effect worth knowing: an application whose only HTTP surface is its health endpoint disappears from the dashboard entirely. One backend service on production does exactly that. Its real work is a Kafka consumer, and an HTTP SLO has nothing to say about it. That is correct, and it needs to be written down, because a blank row otherwise looks like a bug.
Decision two: a 4xx is the caller’s problem
Micrometer classifies each request into an outcome: INFORMATIONAL, SUCCESS, REDIRECTION, CLIENT_ERROR, SERVER_ERROR or UNKNOWN. The SLI counts only SERVER_ERROR as bad. A CLIENT_ERROR is a malformed request, an expired token, a route that does not exist. The service did what it should.
Using Micrometer’s own classification rather than matching on status=~"5.." has one quiet benefit on the Spring Boot versions this fleet runs: requests that abort before a status code is written land in status="UNKNOWN", and Micrometer files those under SERVER_ERROR. A regex on the status label would miss them. Verify this on your own versions before relying on it.
The exclusion has a cost, and the cost is the part to state out loud. Requests the gateway rejects before they reach the application, such as an authorization failure at the edge or a rate limit, never appear in this metric at all. They are neither good nor bad here. They are absent.
Decision three: the window is the calendar month
The service level agreement is written per calendar month. Not per rolling 28 days, not per 30 days. So the dashboard opens with its time range set to now/M through now, month to date, and the SLI panels follow that range through $__range. The quick-range picker carries “this month”, “last month” and “28 days” and nothing else. The 28-day range is there for measurement, not for the SLA: four whole weeks compare like with like across months of different length, which is why every figure in this post is a 28-day or 30-day measurement rather than a calendar month.
The consequence is that zooming the picker changes the SLO number, which surprises people the first time. One row is exempt: the board also carries burn-rate tiles. Burn rate is the error ratio divided by the ratio the budget allows, so 1.0 means the budget is being spent exactly as fast as it accrues and 14.4 means a 30-day budget is gone in about two days. Those tiles are pinned to fixed 1-hour and 6-hour windows and ignore the picker, because a burn rate over a month-to-date window is not a useful alerting signal.
The zero that was not zero
The first version of the ratio had a bug that no test would catch, because the wrong output looks like a healthy dashboard. An application with zero server errors in the window has no SERVER_ERROR series at all. The numerator is not zero. It is an empty set. Dividing an empty set by the total matches nothing, and the application vanishes from the per-application table.
A perfectly healthy service silently disappearing from an availability dashboard is the wrong direction to fail in. The fix substitutes a real zero:
(
sum by (application) (increase(...{outcome="SERVER_ERROR"}[$__range]))
or
(sum by (application) (increase(...[$__range])) * 0)
)
/
sum by (application) (increase(...[$__range]))
The or (total * 0) is load-bearing. Remember it, because it comes back.
What the number cannot see
Everything above is application-tier. The application counts requests it received. It cannot count requests it did not receive, and a gateway answers plenty of requests on its own: a 503 when no healthy upstream exists, an authorization rejection, a rate-limit response. Those never reach the pod.
The uncomfortable consequence: if every replica of a service is down, the gateway answers every request with a 503, and the application counts zero requests and zero errors. Per application that is the empty-set case again, so the service vanishes from the table. The fleet ratio is then computed over the services still standing, and the headline tile at the top does not move. A total outage looks like a quiet night.
The request-based board handles this two ways, both partial. The request-rate tile maps zero traffic to an explicit “NO TRAFFIC” state instead of showing a green zero. And an “edge cross-check” row plots the gateway’s own request and 5xx counters next to the application’s, so edge traffic without matching application traffic is visible as a shape mismatch. Neither turns the gap into a number. I wrote that down in the generator docstring as a known blind spot, shipped the dashboard, and measured the fleet with it.
The measured result, production clusters, 28 days: roughly 470 million requests and about 7,900 server errors, which is 99.998% and, computed from the counts, about 1.7% of a 99.9% error budget spent. It is a true number for what it measures. It is also exactly the number a fleet with one dead service would produce.
The time-based board: time as the denominator
The fix is not a better filter. It is a different denominator. A request-based SLI asks “of the requests we served, how many did we serve badly?” A dead service served none, so the question has no answer. An SLA does not ask that question anyway. It says “99.9% uptime”, which is 43.2 minutes in a 30-day month, and it wants to know how many minutes were down.
So the second board scores time. The window is cut into five-minute slices, and the denominator is the count of slices on the wall clock, computed from a constant rather than from any application’s own series:
count_over_time(vector(1)[$__range:5m])
That expression does not know or care whether the application existed. Absence is downtime by construction.
A slice is good when the application had at least one live instance during it. Liveness comes from process_uptime_seconds, which every service emitting the HTTP family also emits, with the same cluster and application labels, so no join against pod metadata is needed and the Kafka consumer with no HTTP surface gets scored too.
The board carries two SLIs side by side. The first counts a slice bad only when no instance is live. The second also counts it bad when more than 1% of the slice’s requests, probe URIs excluded, were server errors:
# 1 while at least one instance is live, absent otherwise
live = clamp_max(count by (cluster, application) (process_uptime_seconds), 1)
# present only for slices whose server-error ratio breaches the budget
errored = (
sum by (cluster, application) (rate(...{outcome="SERVER_ERROR"}[5m]))
/ sum by (cluster, application) (rate(...[5m]))
) > 0.01
good = live unless errored
uptime = 100 * sum_over_time(good[$__range:5m]) / count_over_time(vector(1)[$__range:5m])
Both SLIs are computed per application per cluster. The headline tiles then take the minimum across the selected clusters, so the number at the top is the worst application in view, not an average that lets a small dead service hide behind a large healthy one.
Over 30 days on production the two SLIs disagreed on 5 of the 32 application-and-cluster pairs. The disagreement is the point. One customer-facing web endpoint scored a flawless 100.0000% on liveness alone while spending 20 minutes above the 1% error threshold on the stricter SLI, about 46% of that 43.2-minute budget. Neither number is wrong. They answer different questions, and the SLA is written in the language of the second one.
Absence-as-downtime has a sharp edge. A cluster onboarded partway through the window scores about 69% for the month, because it did not exist for nearly a third of it. The table splits “down” from “errored” so that all-down and zero-errored reads as “not deployed” rather than as an outage. That is a caption, not a fix, and it needs to be there.
The same bug, one level up
Remember or (total * 0). Three weeks after the time-based board shipped, a review of the board found the empty-set bug again, one level up. sum_over_time(good[$__range:5m]) over an application with no live sample anywhere in the window returns no series, not zero.
The headline tiles wrap every application in min(), and an empty series is not a participant in min(). So an application two hours into a hard outage stopped being counted, and the tiles reported the best surviving application instead. Confirmed against the live store: the numerator came back as an empty vector, and the panel’s “no value” fallback rendered it as a calm 0.00 during a total outage.
The fix has the same shape as before. OR in a zero-valued skeleton series for every application seen at any point in the range, built from max_over_time(process_uptime_seconds[$__range]) multiplied by zero, so a fully absent application holds its row at zero good slices. And drop the “no value” fallback, so an empty result reads as no data rather than as a healthy zero.
The lesson I would like to claim is “check every aggregation for empty-set behaviour”. The honest one is that I knew this bug class well enough to write a docstring about it in July and shipped it again in August. Empty-set semantics in PromQL are a place where the same person makes the same mistake twice. What caught it, in that review, was a generator assertion plus running every one of the board’s 20 panel queries against the live store, not intuition.
What it cost, and what is still not counted
Costs, stated next to the win:
- Every panel query runs about one second over a 30-day range. Fine for a dashboard someone looks at. Not fine for alerting, which is why burn-rate alerting is not built; it belongs in recording rules once the definitions stop moving.
- Liveness carries a roughly five-minute staleness grace from the store’s lookback delta, which matches the five-minute slice. A single missed scrape is not downtime, and a real outage registers within one slice. Short blips are under-reported. Nothing is over-reported.
- Both boards are generated by a Python script that asserts grid non-overlap, balanced query parentheses, declared variables, and that every slice ratio divides by the wall-clock count rather than by the application’s own series. The JSON is committed next to the generator and imported into Grafana by hand, because the central Grafana has no dashboard provisioning. That is a known gap, not a preference.
- No latency SLO. The busiest customer-facing services ship count, sum and max but no histogram buckets, and a mean hides exactly the tail an SLA is written about. One Spring property fixes it per service. The latency row is labelled diagnostic, not SLI, for the services that lack it.
And the blind spot that survives both boards: requests the gateway answered itself. A 503 for no healthy upstream, an edge authorization failure, a rate limit. The time-based board catches the pod-down case that produces most of those 503s, but the gateway-local reply itself is counted nowhere. The real answer is an external blackbox probe hitting the public endpoint the way a customer does. On this fleet, that is not built.
On my own cluster at home it is, in the form of Gatus probing every route continuously, and having it made one thing obvious. The moment the probe existed, the platform had a dependency on the prober, and a config error in the prober became an outage of the monitoring. There is no free exclusion. Something is always left out, and the job is to know what.