# LLM Eval Tooling: DeepEval vs Ragas vs LangSmith vs Cloud

*Published 2026-06-09 · Dmitry Shirokov · shirokoff.ca/blog/llm-evaluation-frameworks · as of June 2026*

A team told me proudly that they had evals. They did — in three places. Ragas scores in an unreproducible six-week-old notebook, 400 unlabeled LangSmith traces, and a spreadsheet of 40 questions last edited in March. None of it wrong; none of it wired to a decision. That's the 2026 failure mode: no path from a metric to a go/no-go. Companion to [evaluating LLM and agent systems](https://shirokoff.ca/blog/evaluating-llm-agent-systems) (the method); this is the tooling bake-off.

## The five problems all this tooling exists for

1. **No `assert output == expected`** — many phrasings are right, output is non-deterministic, quality is multi-dimensional. So you score, which needs a scorer library.
2. **The scorer is a model** — position bias, verbosity bias, self-preference, drift on judge upgrade. So you must evaluate the judge.
3. **Agents fail in the trajectory** — a right answer via four wrong tool calls, $2 and 40s, is a failure at scale. Needs trace- and tool-level scoring.
4. **Offline sets go stale** — your golden set is the questions you thought of; production sends the others. Needs production sampling back into the dataset.
5. **Evals nobody runs are worth nothing** — the manual eval loses to the deadline. Needs CI, or a sampler that runs unattended.

## The three-layer model

Every tool is a subset of: **(1) metric definitions**, **(2) the runner** (pytest/CI, production sampler, batch), **(3) the store + surface** (trends, alerts, triage). Most "which framework" arguments compare tools from different layers. **Choose layer 3 first** — where your traces already live — and 1 and 2 follow. Picking a metric library that can't see production is the classic mistake.

## Metrics worth their cost

| Metric | Catches | Gotcha |
|---|---|---|
| Deterministic checks (schema, required field, forbidden string, tool called) | The dumb failures behind most incidents | Skipped for feeling too simple; catch more than any judge |
| Exact match / F1 / regex | Classification, extraction, routing | Useless for open-ended generation |
| Faithfulness / groundedness | Claims the context doesn't support — the #1 RAG failure | Needs retrieved context logged; teams forget |
| Context precision / recall | Whether retrieval got the right chunks | Recall needs labeled relevant docs — pay once |
| Answer relevance | Fluent answers to a question nobody asked | Weak correlation with satisfaction alone |
| Intent resolution | Agent grasped the request | Rewards confident restatement; pair with adherence |
| Task adherence / goal success | Job done end to end within instructions | Needs "the job" defined per case |
| Tool selection + parameter accuracy | The two agent failures that cost money | Needs trace data; I/O-only harnesses can't see it |
| Safety: harmfulness, stereotyping, jailbreak, PII leakage | The failures that reach a regulator | Needs adversarial inputs — that's red-teaming |
| Cost/task, latency, tool-call count, refusal rate | The agent "working" while looping | Not quality; most under-monitored |

Two nobody tracks: **refusal rate** is the best canary for a system-prompt regression (it moves before quality scores), and **tool-call count per task** catches the agent that started taking nine steps for a three-step job after a model upgrade — a cost incident dressed as a working system.

## OSS frameworks

**DeepEval** — pytest-native: test cases, thresholds, non-zero exit code. That exit code is the value: an eval becomes a CI gate with no glue. Broad metric library, **G-Eval** (rubric → judged metric), and **DAG metrics** — judgment as a decision tree of small deterministic questions, which cuts judge variance hard, because "does it cite a source? → does the source contain the claim?" is far more stable than "rate faithfulness 1–5."

**Ragas** — a focused RAG metrics library (faithfulness, answer relevancy, context precision/recall, plus agent and multi-turn metrics), the most carefully thought-through RAG metrics available. Deliberately not a platform: no UI, no tracking, no monitoring. Caveat: judge variance — fix model, temperature, prompt version and read it as a trend line, not a grade.

**LangSmith** — observability platform where eval is one feature: traces → datasets → experiments → annotation queues → online evaluators. Its real strength is the loop: a bad production trace becomes a permanent regression case in two clicks. Best if you're already on LangChain/LangGraph.

Also: **Promptfoo** (declarative YAML matrices + red team; fastest "can the cheaper model do this?"), **Arize Phoenix** (OTel-native, self-hostable, strong trace debugging), **MLflow** (one lineage story if you already run it for classical ML), **Braintrust** (polished experiment/dataset layer; pairs with a pytest gate).

## Azure — Microsoft Foundry evaluation

The `azure-ai-evaluation` SDK plus the Foundry portal. Distinguishing feature: **safety evaluation is a hosted service**, not a prompt you maintain.

- **Quality/RAG**: groundedness, relevance, retrieval, coherence, fluency, similarity.
- **Agent**: `IntentResolutionEvaluator`, `ToolCallAccuracyEvaluator`, `TaskAdherenceEvaluator` — plus relevance and groundedness — supported through **agent converters** that turn a Foundry Agent Service thread or Semantic Kernel run into evaluator input. The converter is the real time-saver.
- **Safety/adversarial**: hateful-unfair, violence, self-harm, sexual, protected material, code vulnerability, jailbreak — via an Azure-hosted safety service. Plus an **AI Red Teaming Agent** reporting attack success rate by risk category and strategy.

Run locally with `evaluate()` over JSONL, push results to a Foundry project for the portal, and configure **continuous evaluation** so sampled live runs score into Application Insights where alert rules fire. Compelling when your agents already run in Foundry — the traces are there. Pin the judge deployment *and* API version.

## AWS — Bedrock AgentCore Evaluations

**GA March 31, 2026.** Managed, and framework-agnostic by design: it ingests **OpenTelemetry / OpenInference** traces (Strands, LangGraph called out), normalizes them, and scores with LLM-as-judge. Your agent emits traces instead of importing an eval SDK.

- **Built-ins**: correctness, faithfulness, helpfulness, harmfulness, stereotyping, tool selection accuracy, tool parameter accuracy — referenced as `Builtin.Helpfulness`. Immutable (model + prompt fixed): flexibility traded for comparability.
- **Levels**: session, trace, or **tool call**. Tool-level scoring localizes agent failure to a specific tool — no I/O harness can do that.
- **Modes**: online (sampled live traffic), on-demand (recent window), batch, dataset, and simulation.
- **Custom evaluators**: LLM-as-judge (your model/instructions/rating scale) or **code-based via Lambda** for deterministic business rules — the option teams overlook then reinvent.
- **Results stream to CloudWatch** in the AgentCore Observability dashboard, so alarms are your existing mechanism. GA quotas: 1,000 eval configs per region, 100 active.

Naming trap: AgentCore Evaluations (agent behaviour over traces) is **not** Bedrock model evaluation (which model / which knowledge base). Different services, similar console words.

CLI shape: `agentcore add evaluator --level SESSION|TRACE|TOOL`, `agentcore run eval --days 7`, `agentcore add online-eval --sampling-rate 10`, `agentcore evals history`.

## Side by side

| Tool | Shape | Best at | Weak at |
|---|---|---|---|
| DeepEval | pytest library (+ platform) | CI gating; DAG metrics for stable judging | Production monitoring alone |
| Ragas | Metrics library only | Rigorous RAG diagnosis | No UI/tracking; judge variance |
| LangSmith | Tracing + datasets + online eval | Trace → dataset → regression loop | Vendor gravity off LangChain |
| Promptfoo | YAML matrices + red team | Prompt/model bake-offs | Deep trajectory analysis |
| Phoenix | OTel-native, self-hostable | Trace debugging, own the data | Less turnkey |
| Foundry eval | SDK + portal + continuous eval | Hosted safety, agent converters, App Insights alerts | Azure-shaped; inherited judges |
| AgentCore Evaluations | Managed service over OTel traces | Session/trace/**tool** scoring, CloudWatch, Lambda evaluators | Immutable built-ins; AWS-shaped |

## How to choose

Azure-native agents: Foundry continuous evaluation + hosted safety for production, DeepEval in pytest for the merge gate. AWS-native: AgentCore online sampling + a couple of Lambda code evaluators, DeepEval/Promptfoo in CI, and use tool-level evaluation early. Multi-cloud: Ragas/DeepEval metrics, your runner, OTel into Phoenix. RAG-heavy and struggling: Ragas on a labeled retrieval set before touching prompts or models.

**Two runners, one dataset.** OSS library gating CI, platform sampling production, and a production failure becomes a CI case the same day. The shared dataset is the integration point.

## The failure mode that will get you

Your judge moves and your baseline moves with it. A judged metric is an instrument made of a model, a prompt, and a temperature; change any (including a silent provider upgrade behind an unpinned alias) and history is incomparable. I watched a team hunt a "quality regression" for a week that was a judge upgrade. Defences: **pin the judge version** and treat changes as a re-baseline migration; keep a **human-labeled golden set** and measure judge-to-human agreement; prefer **decomposed judgments** over holistic 1–5 scores. And sample production — judge tokens on 100% of traffic can cost more than the traffic.

## Carry-away

Pick the store first: the evaluator that can read production without a plumbing project wins on effort. Foundry if your agents live there; AgentCore for the sharpest diagnostic (tool-level scoring); OTel + OSS if you want portability. Then two runners over one dataset, deterministic checks before judged metrics, refusal rate and tool-call count tracked alongside quality, judge pinned and audited. An eval that blocks nothing is a dashboard — wire one metric to one decision, then grow.

*Related: [Evaluating LLM and agent systems](https://shirokoff.ca/blog/evaluating-llm-agent-systems) · [LLM observability](https://shirokoff.ca/blog/llm-observability) · [Agent harness design](https://shirokoff.ca/blog/agent-harness-design) · [LLM FinOps and token costs](https://shirokoff.ca/blog/llm-ai-finops-token-costs) · [RAG on Azure](https://shirokoff.ca/blog/rag-on-azure) · [MLflow tracking and registry](https://shirokoff.ca/blog/mlflow-experiment-tracking-registry)*