Data quality SLOs: error budgets for pipelines

The channel had four thousand unread messages. Every one of them was a data test result, every one had fired at severity error, and the team had learned — correctly, adaptively — to ignore all of them. When a genuinely broken dataset went out to a regulator-facing report, the alert was in there. It was message 3,847.

The instinct after an incident like that is to add more tests. That makes it worse. The problem was never coverage; it was that nobody had ever decided what "broken" means for each dataset, so every deviation was treated as equally urgent, which is the same as treating none of them as urgent.

SRE solved this shape of problem for services twenty years ago, and the machinery ports over almost unchanged: pick a small number of indicators, set a target you can defend, and derive an error budget that tells you when to stop shipping features and fix the thing. This is how I apply that to data.

The vocabulary, in data terms

The word doing the heavy lifting is agreed. An SLO invented by the data team is a wish; an SLO the consuming team signed off on is a contract, and the difference shows up the first time you have to say "this is within budget, we'll fix it Monday" and have that be an acceptable answer rather than an argument.

Four indicators cover most of it

Resist the urge to invent twenty. Four SLIs describe almost every way a dataset disappoints someone:

• Freshness — how old is the newest data? The one users notice first, and the one that most often matters more than correctness. A slightly wrong number now usually beats a perfect number tomorrow, in most (not all) contexts.
• Completeness — did all the rows arrive? Measured as volume against an expectation, ideally a seasonal one, because "fewer rows than yesterday" is meaningless on a Monday.
• Validity — do the rows satisfy the rules that must hold? Nulls in required fields, broken referential integrity, values outside a domain. This is where your existing tests live.
• Consistency — does it agree with the other place this number exists? The cross-system reconciliation that catches the silent failures, and the check teams skip because it's the most work.

Note what's missing: accuracy in the philosophical sense. You usually can't measure whether the data reflects reality without a second source of truth — which is exactly what the consistency SLI approximates. Don't promise an SLO you have no instrument for.

Tiering, so the four don't become four hundred

Not every dataset deserves an SLO, and pretending otherwise is how you end up back at four thousand unread messages. Tier by consequence:

The honest count in most organizations is single-digit Tier 1, a few dozen Tier 2, and hundreds of Tier 3. If your tiering doesn't look roughly like that, you've tiered by who complained loudest rather than by consequence — and the fix is to ask each claimant what happens, concretely, if the dataset is four hours late.

graph TB
  DS[("Dataset")] --> TIER{"Tier by consequence"}
  TIER -->|"Tier 1"| S1["Freshness · Completeness
Validity · Consistency"]
  TIER -->|"Tier 2"| S2["Freshness · Completeness
Validity"]
  TIER -->|"Tier 3"| S3["Freshness, or nothing"]
  S1 --> MEAS["Measure continuously
emit as metrics, not log lines"]
  S2 --> MEAS
  S3 --> MEAS
  MEAS --> BUDGET{"Error budget
consumed this window"}
  BUDGET -->|"burning fast
(half the budget in a day)"| PAGE["Page — something is broken now"]
  BUDGET -->|"burning slow
but will exhaust"| TICKET["Ticket — fix this sprint"]
  BUDGET -->|"exhausted"| FREEZE["Stop feature work on this pipeline
until reliability is restored"]
  BUDGET -->|"healthy"| SHIP["Ship features"]
  FREEZE -.->|"the conversation the budget
is designed to make automatic"| SPONSOR["Sponsor agrees, because
they signed the SLO"]

Setting targets you can defend

Three rules keep targets honest.

Measure first, promise second. Instrument the SLI, watch it for a month, then set the target near where you already are — slightly tighter if you need improvement. Targets pulled from the air are either trivially met (and therefore uninformative) or permanently breached (and therefore ignored).

Never target 100%. A perfect target has no error budget, which means every blip is an incident and the framework collapses back into the alert channel you were escaping. If a consumer insists on 100%, that's a genuine architecture conversation about redundancy and cost, not a monitoring configuration.

Scope the window to the decision. "Fresh within 60 minutes" means something different at 3am than at 9am if nobody looks at 3am. Business-hours SLOs are usually more honest and much cheaper to meet, and the honesty is worth more than the round number.

# SLOs as data, in the repo, next to the pipeline. Reviewed in the PR that
# changes them, and — the part that matters — signed off by the consumer.
dataset: finance.daily_revenue
tier: 1
owner: data-platform@example.com
consumers: [finance-reporting, exec-dashboard]
approved_by: cfo-office              # an SLO nobody agreed to is a wish
slos:
  - sli: freshness
    definition: "minutes between source transaction time and availability"
    target: "p99 <= 90 minutes"
    window: "business hours, rolling 30 days"
    budget: "1% of business hours"

  - sli: completeness
    definition: "row count vs 28-day seasonal expectation"
    target: ">= 98% of expected, 99.5% of days"
    window: "rolling 30 days"

  - sli: validity
    definition: "rows passing required-field and domain rules"
    target: ">= 99.9% of rows"
    window: "rolling 30 days"

  - sli: consistency
    definition: "daily total vs the general ledger control total"
    target: "abs(delta) <= 0.1%, 100% of days"     # tier 1: no budget here
    window: "rolling 30 days"

alerting:
  fast_burn:  { threshold: "50% of budget in 24h", action: page }
  slow_burn:  { threshold: "budget exhausted in < 7d at current rate", action: ticket }
  exhausted:  { action: "freeze feature work on this pipeline" }

Where existing tests fit

You almost certainly already have tests — dbt tests, expectations, custom checks — and this doesn't replace them. It reorganizes them. Tests are how you measure the validity SLI, and possibly completeness; the SLO layer is what decides which failures matter and who gets woken.

The practical change is that a test's severity stops being a property of the test and becomes a property of the dataset's tier. The same null-check on a Tier 3 staging table writes a metric; on a Tier 1 finance table it burns budget and can page. That single reframing eliminated most of the noise in the team I opened with, and they didn't delete a single test.

One more piece worth connecting: an SLO is the operational half of a data contract. The contract says what the shape and semantics are; the SLO says how reliably it will be there. A producer who has agreed to both is a producer you can build on, which is the relationship these two artifacts exist to create — and it also gives the pipeline's on-call a defensible answer at 3am, which is the theme of data pipeline on-call operations.

What to carry away

More tests do not fix alert fatigue; deciding what "broken" means does. Borrow the SRE machinery: SLIs (freshness, completeness, validity, consistency — four cover almost everything), SLOs that a named consumer actually agreed to, and an error budget that converts monitoring into a decision.

Tier by consequence so the small number of datasets that can hurt someone get the full treatment and the hundreds that can't get a dashboard. Measure for a month before promising a target, never target 100%, and scope the window to when the decision actually gets made.

Then wire the budget to something real: alert on burn rate rather than on every SLI breach, and freeze feature work on a pipeline whose budget is exhausted. Keep the SLOs as data in the repo, reviewed in a pull request, with the consumer's sign-off in the file — because the sign-off is what makes "we're within budget, we'll fix it Monday" an acceptable sentence rather than the start of an argument.