Every team I've worked with builds their first eval set the same way. Someone opens a spreadsheet on a Friday afternoon, types forty questions they think users might ask, writes what a good answer looks like, and the team runs against it for the next six months. It is infinitely better than nothing. It is also, in a specific and predictable way, a lie — because those forty questions were written by someone who knows how the system works, in the phrasing the system handles best, about the topics the team already thought about.
The gap shows up as a peculiar pattern: the eval score stays flat and high while user complaints climb. Nothing is broken, exactly. The dataset just never contained the questions people actually ask — the ones with typos, the ones that assume context from three messages ago, the ones about the edge of your document corpus where coverage is thin.
An eval is a dataset plus a scorer, and the dataset is where the information lives. Scorers are plumbing you can swap; the dataset encodes what you have decided "good" means, and building it from real traffic is the single highest-value thing you can do for evaluation quality. This is how I build one, what it costs, and how to keep it from rotting.
What a golden set actually needs to be
Three properties, and they pull against each other, which is why this takes judgment rather than a script.
- Representative — the mix of cases mirrors real usage, so a score movement means something about production rather than about a corner of it.
- Discriminating — it contains cases your system currently gets wrong or nearly wrong. A dataset everything passes has zero information content, no matter how large.
- Stable — the same case means the same thing next quarter, so scores are comparable over time. This is what "golden" means, and it's why you version it.
Representative and discriminating are in tension: real traffic is mostly easy, so a purely random sample will be dominated by cases everything handles. The resolution is deliberate stratification — sample enough easy cases to keep the mix honest, then over-sample the hard strata on purpose and weight them when reporting.
Where the cases come from
Five sources, in the order I'd exploit them. The first two cost nothing and are worth more than the rest combined.
| Source | What it gives you | Cost |
|---|---|---|
| Thumbs-down and complaints | Cases you are provably wrong on, already identified by a human | Nearly free — you have to be capturing them |
| Escalations and handoffs | Where the system gave up: the boundary of your competence | Free if you log the handoff reason |
| Stratified random production sample | The honest mix; catches what nobody thought to test | Labeling time, which is the real budget line |
| Uncertainty mining — low retrieval scores, hedged answers, high step counts | Cases at the edge of the system's competence, before users notice | Cheap; needs your traces to carry the signals |
| Hand-written adversarial cases | Policy and safety behaviour you must never regress on | Expensive per case, and worth it for a small set |
Uncertainty mining is the one teams skip and shouldn't. If your traces record retrieval scores, refusal or hedging language, and step counts, you can find the questions your system nearly failed — which are exactly the cases most likely to fail after your next change, and which no human would have thought to write.
graph TB
PROD[("Production traces")] --> STRAT["Stratify
intent · tenant type · length · retrieval score"]
STRAT --> SAMPLE["Sample per stratum
enough easy, deliberate hard"]
FEEDBACK[("Thumbs-down
escalations")] --> POOL["Candidate pool"]
MINE["Uncertainty mining
low scores, hedges, long loops"] --> POOL
SAMPLE --> POOL
POOL --> SCRUB["Scrub
redact PII, keep the shape"]
SCRUB --> LABEL["Label
reference answer + must/must-not"]
LABEL --> REVIEW{"Second reviewer
on a sample"}
REVIEW -->|"disagreement > threshold"| RUBRIC["Fix the rubric, not the labeller"]
RUBRIC --> LABEL
REVIEW -->|"agreed"| GOLD[("Golden set v-N
versioned, in git")]
GOLD --> CI["CI gate"]
GOLD --> DRIFT["Quarterly refresh
retire solved, add new failures"]
DRIFT --> POOL
The loop, with the two steps everyone underestimates highlighted by their position: scrubbing sits before labelling, because you do not want annotators reading raw customer data; and the disagreement branch fixes the rubric rather than the annotator, since two careful people disagreeing means your definition of "good" is underspecified — which will also confuse an LLM judge.
The labeling economics nobody budgets
This is where these projects die, so put real numbers on it before you start. A domain expert labeling a RAG case properly — read the question, read the retrieved context, decide whether the answer is supported, write a reference answer or an acceptance rubric — takes somewhere between three and ten minutes depending on domain. Two hundred cases is therefore roughly one to three days of an expert's time, plus review. That is a completely reasonable investment and a completely unreasonable surprise.
Three things that cut the cost materially:
- Label acceptance criteria, not perfect answers. Writing the ideal response is slow and mostly wasted, since you'll score with a judge against criteria anyway. "Must state the 30-day window; must cite the refund policy document; must not promise an exception" is faster to write, faster to review, and produces a more stable score than free-text similarity.
- Pre-fill with the current system's output. Give the annotator the existing answer and ask them to accept, correct, or reject. Editing beats authoring by a wide margin — with one caution worth naming: it anchors people toward accepting, so make "reject" cheap and review a sample of the accepts.
- Draft with a model, verify with a human. A strong model can propose criteria for a case; the expert verifies in under a minute. This works well and is the standard move — as long as the human genuinely verifies. An unverified model-labeled dataset measures your judge's agreement with itself, which is a number that will make you feel good and tell you nothing.
Size: smaller than you think, then grow it
Teams either build twelve cases or plan for five thousand and never finish. For a first production gate, 150–300 well-chosen cases is a genuinely useful set if the strata are right — enough to detect a real regression, small enough to run in CI in minutes and cheap enough to re-label when your definition of good changes. Grow it by adding failures as they arrive, not by bulk-sampling more of what you already pass.
PII, and why scrubbing comes before labeling
Production traffic is customer data, which makes the golden set a compliance object rather than a test fixture. Three rules I'd defend anywhere:
Redact before a human ever reads it. Run detection and replacement on the way into the candidate pool, not after annotation. Otherwise every annotator has read raw customer records and you have quietly widened your data-access surface to include a spreadsheet.
Preserve the shape, replace the value. This is the part done badly most often. Replacing a customer name with [REDACTED] destroys the case — the model's behaviour on "why was Acme Corp's claim denied" is not the same as on "why was [REDACTED]'s claim denied". Substitute realistic surrogates instead: a consistent fake company, a valid-format policy number, a plausible date. The case stays a case.
Treat the golden set as a governed asset. It lives in a repo with access control, it has a retention policy, and it's in your data inventory. It is derived customer data, and it will outlive the incident that motivated it — the same principle that governs any derived copy of customer data applies with full force here.
# A case, as I'd store it: JSONL in git, one object per line, versioned with the
# code that consumes it. Criteria rather than a perfect answer; provenance so a
# disputed label can be traced back to the run it came from.
{
"id": "case-0142",
"added": "2025-06-30",
"source": "thumbs_down", # feedback | escalation | sample | mined | adversarial
"stratum": "policy_edge", # what this case is here to test
"input": "we cancelled on day 31, is there really nothing you can do",
"context_ref": "trace:9f2c…", # the production run this came from
"must_include": [
"states the 30-day refund window",
"offers the escalation path to a human"
],
"must_not_include": [
"promises an exception",
"invents a policy clause"
],
"reference": "The standard refund window is 30 days from purchase…",
"labeler": "dana", "reviewed_by": "sam",
"weight": 3.0 # over-sampled stratum, down-weight when reporting
}Keeping it honest over time
A golden set decays in two directions and both are quiet.
It gets too easy. You fix the failures it contains, and afterwards everything passes forever — a green suite with no information in it. The fix is a standing rule: every production failure becomes a case, the same week. That single habit does more for evaluation quality than any tooling decision, and it's the reason to make adding a case a two-minute operation rather than a ticket.
It goes stale. Your product changes, your policies change, and cases that encoded last year's correct answer now fail for the right reason. Re-validate the set quarterly, retire cases whose intent no longer exists, and treat a change to a reference answer as a versioned event with a note — because a score that moves because the dataset changed, silently, is worse than no score at all.
Version the set alongside the code, report which version a score came from, and never compare scores across versions without saying so.
Three ways a golden set quietly lies. Contamination. Cases that made it into your prompt as few-shot examples, or into a fine-tuning run, are no longer measuring generalization — they're measuring memorization, and the score goes up while the product doesn't. Keep a strict wall: examples come from a separate pool, never from the eval set. Selection bias from feedback. Thumbs-down data is invaluable and skewed — a specific slice of users complains, about a specific class of failures. If your set is mostly complaints, you'll optimize hard for a loud minority and regress on the silent majority. Blend it with a stratified random sample and report both. The single-annotator rubric. One person labeling alone produces a set that encodes their interpretation of "good" — and every later disagreement about whether the system is improving becomes an argument about their judgment. Have a second person label a sample of 20–30 cases; if agreement is poor, the rubric is underspecified, and that ambiguity will hurt your LLM judge exactly as much as it hurt your humans.
What to carry away
Your eval quality is bounded by your dataset, and forty invented questions bound it low. Build the set from traffic you already have: thumbs-down and escalations first (free, and provably real failures), then a stratified sample for an honest mix, then uncertainty mining for the cases at the edge of competence that nobody would think to write.
Budget the labeling honestly — three to ten minutes per case with a domain expert — and cut it by labeling acceptance criteria rather than perfect answers, pre-filling with current output, and drafting with a model that a human genuinely verifies. Aim for 150–300 well-stratified cases for a first gate, then grow by adding failures rather than bulk-sampling passes.
Scrub before humans read, and preserve the shape of what you replace. Version the set in git, keep a rule that every production failure becomes a case that week, re-validate quarterly, and watch for the three quiet lies: contamination with your prompt examples, selection bias toward complainers, and a rubric only one person has ever interpreted. A dataset built this way is the most durable asset in your AI stack — models change, frameworks change, and it keeps telling you the truth about all of them.