# Building a Golden Eval Dataset From Production Traffic

*Published 2025-07-08 · Dmitry Shirokov · shirokoff.ca/blog/golden-eval-datasets-from-production*

Every team builds its first eval set the same way: someone types forty questions into a spreadsheet on a Friday. Better than nothing, and a specific lie — those questions were written by someone who knows how the system works, in the phrasing it handles best, about topics the team already considered. The tell is a flat, high eval score while complaints climb. **An eval is a dataset plus a scorer, and the dataset is where the information lives.**

## Three properties, in tension

**Representative** (mirrors real usage), **discriminating** (contains cases you get wrong — a set everything passes has zero information), **stable** (same case means the same thing next quarter). The first two pull apart, because real traffic is mostly easy. Resolve it with deliberate **stratification**: enough easy cases to keep the mix honest, over-sample hard strata, weight when reporting.

## Where cases come from

| Source | Gives you | Cost |
|---|---|---|
| **Thumbs-down / complaints** | Provable failures, already human-identified | ~Free |
| **Escalations / handoffs** | The boundary of your competence | Free if you log the reason |
| **Stratified random sample** | The honest mix; what nobody thought to test | Labeling time |
| **Uncertainty mining** (low retrieval scores, hedged answers, long loops) | Cases at the edge, before users notice | Cheap — needs traces |
| **Hand-written adversarial** | Policy/safety behaviour you must never regress | Expensive per case, worth a small set |

Uncertainty mining is the skipped one: the questions your system *nearly* failed are exactly what breaks after your next change, and no human would think to write them.

## Labeling economics

Three to ten minutes per RAG case for a domain expert. Two hundred cases ≈ one to three expert-days plus review — reasonable as an investment, unreasonable as a surprise. Cut it three ways: **label acceptance criteria, not perfect answers** ("must state the 30-day window; must not promise an exception" is faster to write, faster to review, more stable to score); **pre-fill with current output** so annotators edit rather than author (caution: it anchors toward accepting, so make reject cheap and review a sample); **draft with a model, verify with a human** — genuinely verify, or you're measuring your judge's agreement with itself.

Size: **150–300 well-stratified cases** is a genuinely useful first gate — enough to catch a regression, cheap enough to re-label when "good" changes. Grow by adding failures, not by bulk-sampling more passes.

## PII: scrub before labeling

**Redact before a human reads it** — on the way into the candidate pool, not after annotation, or every annotator has read raw customer records. **Preserve the shape, replace the value**: `[REDACTED]` destroys the case; substitute realistic surrogates (consistent fake company, valid-format policy number, plausible date). **Treat the set as a governed asset** — access control, retention, data inventory. It's derived customer data and it outlives the incident that motivated it.

Store cases as JSONL in git: id, source, stratum, input, `context_ref` back to the production trace, `must_include` / `must_not_include`, reference, labeler + reviewer, stratum weight.

## Keeping it honest

It decays two ways. **Too easy**: you fix what it contains and it goes permanently green. Standing rule — *every production failure becomes a case the same week*, which does more for eval quality than any tooling choice. **Stale**: policies change and cases fail for the right reason. Re-validate quarterly, retire dead intents, treat a changed reference as a versioned event. Version alongside code, report which version a score came from, never compare across versions silently.

## Three quiet lies

**Contamination** — cases that leaked into few-shot examples or fine-tuning measure memorization; keep a strict wall between example pool and eval set. **Selection bias** — thumbs-down data is invaluable and skewed toward a loud minority; blend with a stratified sample and report both. **The single-annotator rubric** — one labeler encodes one interpretation of "good", and every later disagreement becomes an argument about their judgment. Double-label 20–30 cases; poor agreement means the rubric is underspecified, and that ambiguity will hurt your LLM judge exactly as much.

## Carry-away

Eval quality is bounded by the dataset. Build from traffic you have: feedback and escalations first, then a stratified sample, then uncertainty mining. Budget labeling honestly and cut it with criteria-not-answers, pre-filling, and verified model drafts. Scrub before humans read, preserving shape. Version in git, add every production failure that week, re-validate quarterly. Built this way it's the most durable asset in your AI stack — models and frameworks change, and it keeps telling you the truth about all of them.

*Related: [GraphRAG](https://shirokoff.ca/blog/graphrag) · [What MCP standardizes](https://shirokoff.ca/blog/model-context-protocol) · [Building a health data lakehouse on Databricks](https://shirokoff.ca/blog/building-health-data-lakehouse-databricks)*