# Data Quality SLOs: Error Budgets for Pipelines

*Published 2024-12-18 · Dmitry Shirokov · shirokoff.ca/blog/data-quality-slos-error-budgets*

Four thousand unread messages in the channel, every one a data test at severity *error*, and a team that had correctly learned to ignore all of them. When a genuinely broken dataset reached a regulator-facing report, the alert was message 3,847. The instinct afterwards is more tests, which makes it worse. **The problem was never coverage — nobody had decided what "broken" means per dataset, so every deviation was equally urgent, which is the same as none being urgent.**

## The vocabulary, in data terms

| SRE | Data | Example |
|---|---|---|
| **SLI** | A dataset property a consumer feels | Freshness: minutes from source event to availability |
| **SLO** | The agreed threshold | "Fresh within 60 min, 99% of business hours" |
| **Error budget** | Allowed violation | 1% of business hours ≈ 1.7 h/week of staleness |
| **Burn rate** | How fast you're consuming it | Half the monthly budget in a day → page |

The word doing the work is **agreed**. An SLO invented by the data team is a wish; one the consumer signed off on is a contract — the difference appears the first time "this is within budget, we'll fix it Monday" is an acceptable answer instead of an argument.

## Four indicators cover most of it

**Freshness** (noticed first, often matters more than correctness), **completeness** (volume vs a *seasonal* expectation — "fewer rows than yesterday" is meaningless on a Monday), **validity** (required fields, referential integrity, domains — where your existing tests live), **consistency** (agreement with the other place this number exists — the cross-system reconciliation that catches silent failures, skipped because it's the most work). Note what's absent: accuracy in the philosophical sense, which you can't measure without a second source of truth. Don't promise an SLO you have no instrument for.

## Tier, or four SLIs become four hundred

| Tier | What's in it | SLOs | On breach |
|---|---|---|---|
| **1** — regulatory, financial, customer-facing | A handful | All four, tight, consistency required | Page, incident, post-mortem |
| **2** — business-critical internal | Exec dashboards, ops decisions | Freshness, completeness, validity | Ticket in hours; page on fast burn |
| **3** — everything else | Exploration, staging | Freshness or nothing | Dashboard. Nobody wakes up. |

Honest counts: single-digit Tier 1, dozens of Tier 2, hundreds of Tier 3. If yours differs you've tiered by who complained loudest — ask each claimant what concretely happens if the dataset is four hours late.

## Targets you can defend

**Measure first, promise second** — instrument, watch a month, set the target near where you already are. **Never target 100%** — a perfect target has no budget, so every blip is an incident and you're back in the alert channel. **Scope the window to the decision** — business-hours SLOs are more honest and much cheaper than round-the-clock ones.

Keep SLOs as YAML in the repo next to the pipeline: tier, owner, consumers, **`approved_by`**, per-SLI definition/target/window/budget, and alerting rules for fast burn (page), slow burn (ticket), exhausted (**freeze feature work**).

## Two ways this goes wrong

**SLOs as a reporting exercise** — targets written, dashboard built, monthly slide green, nothing changes because no decision is wired to the budget. The freeze is the point; the dashboard is decoration. **Alerting on the SLI instead of the burn rate** — paging every time freshness exceeds 60 minutes wakes you for blips you had budget for. One of those is sustainable at 3am; the other is four thousand unread messages with better vocabulary.

## Where existing tests fit

This reorganizes rather than replaces them: **tests measure the validity SLI**, and the SLO layer decides which failures matter and who gets woken. Severity stops being a property of the test and becomes a property of the dataset's tier — the same null check writes a metric on Tier 3 and burns budget on Tier 1. That reframing removed most of the noise for the team above without deleting a single test.

An SLO is also the operational half of a data contract: the contract says what the shape and semantics are, the SLO says how reliably it'll be there — and it gives on-call a defensible answer at 3am.

*Related: [Data pipeline on-call operations](https://shirokoff.ca/blog/data-pipeline-on-call-operations) · [Designing a data pipeline](https://shirokoff.ca/blog/designing-a-data-pipeline) · [Debezium and CDC](https://shirokoff.ca/blog/debezium-cdc)*