LLM Eval Tooling: DeepEval vs Ragas vs LangSmith vs Cloud

I walked into a project where the team told me, with justified pride, that they had evals. They did. They had them in three places. A notebook with Ragas scores from a run six weeks earlier that nobody could reproduce because the retriever had changed twice since. A LangSmith project holding four hundred production traces, unlabeled, that someone opened when a customer complained. And a spreadsheet of forty hand-written questions with a column of green and red cells, last edited in March.

None of it was wrong. None of it was wired to a decision. That's the actual failure mode with evaluation in 2026 — not an absence of metrics, but an absence of any path from a metric to a go / no-go. So when someone asks me which eval framework to use, the question I answer first is a different one: what decision is this eval going to make, and who gets blocked when it fails?

This is the tooling article. I've already written the one about how to evaluate LLM and agent systems — the offline harness, LLM-as-judge discipline, online evaluation, letting evals gate releases. Read that for the method. This one is the bake-off: what the OSS frameworks each actually are, how the cloud-native offerings work on Azure and AWS, the metrics worth computing, and how I'd choose. Everything as of June 2026.

The problems eval tooling is trying to solve

Every design choice in every one of these tools traces back to one of five problems. Knowing which problem a tool was built for tells you more than any feature matrix.

1. There's no assert output == expected. Many phrasings are correct, output is non-deterministic, and quality is multi-dimensional — an answer can be fluent and wrong, correct and unfaithful to its sources, right and unsafe. So you score instead of comparing, which means you need a scorer library.
2. The scorer is usually a model, and models are unreliable judges. LLM-as-judge brings position bias, verbosity bias, self-preference, and score drift when the judge model is upgraded underneath you. Which means you need to evaluate the judge.
3. Agents fail in the trajectory, not the final answer. A correct answer reached by calling the wrong tool four times, burning $2 and 40 seconds, is a failure you'll pay for at scale. Which means you need trace-level and tool-level scoring, not just input/output scoring.
4. Offline sets go stale. Your golden dataset represents the questions you thought of in March; production sends you the ones you didn't. Which means you need production sampling feeding back into the dataset.
5. Evals nobody runs are worth nothing. The manual eval always loses to the deadline. Which means the eval has to live in CI, or in a sampler that runs whether or not anyone is looking.

The three-layer model that makes the choice obvious

Here's the framing I use to cut through the marketing, because every tool in this space is some subset of three layers: the metric definitions, the runner that executes them, and the store that keeps results and shows you trends. Nearly every "which framework" argument is actually people comparing tools that occupy different layers.

graph TD
  subgraph M["Layer 1 — Metrics (what 'good' means)"]
    OSS1["Ragas · DeepEval metrics
G-Eval · custom judges"]
    CLOUD1["Built-in evaluators
Foundry · AgentCore"]
  end
  subgraph R["Layer 2 — Runner (when + on what)"]
    CI["pytest / CI gate
on every PR"]
    SAMP["Production sampler
N% of live traffic"]
    BATCH["Batch / on-demand
over a dataset or window"]
  end
  subgraph S["Layer 3 — Store + surface (trends, alerts, triage)"]
    PLAT["LangSmith · Confident AI
Braintrust · Phoenix"]
    NATIVE["CloudWatch dashboards
Foundry portal + App Insights"]
  end
  OSS1 --> CI
  OSS1 --> BATCH
  CLOUD1 --> SAMP
  CLOUD1 --> BATCH
  CI --> PLAT
  SAMP --> NATIVE
  BATCH --> PLAT
  BATCH --> NATIVE
  DEC{"Where do your
traces already live?"} -.->|"decides layer 3 first"| S

Which metrics actually earn their cost?

Use the cheapest scorer that captures what you care about, and reach for a judge only when nothing simpler will do. The families, roughly in order of how much I trust them:

Two of those deserve emphasis because almost nobody tracks them. Refusal rate is the best single canary for a system-prompt regression — it moves before quality scores do. And tool-call count per task is how you catch the agent that started taking nine steps to do a three-step job after a model upgrade, which is a cost incident dressed as a working system.

The OSS frameworks: what each one actually is

DeepEval — evals as unit tests

DeepEval is a pytest-native evaluation framework: you write test cases, assert metrics against thresholds, and get pass/fail with a non-zero exit code. That last detail is the whole value proposition — it makes an eval into a CI gate without any glue code. It ships a broad metric library (RAG metrics, agent metrics, safety metrics), G-Eval for defining a judged metric from a plain-language rubric, and — the feature I've come to like most — DAG metrics, where you express the judgment as a decision tree of small deterministic questions instead of one big vibe check. That structure dramatically reduces judge variance, because "does the answer cite a source? → yes/no → does the cited source contain the claim? → yes/no" is a far more stable question than "rate the faithfulness from 1 to 5."

Where it fits: your CI pipeline, as the thing that blocks the merge. It has a hosted companion platform (Confident AI) if you want the dashboards, but the library alone is genuinely useful standalone.

# DeepEval as a release gate: a failing metric fails the build. That's the point.
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric, GEval
from deepeval.test_case import LLMTestCaseParams

JUDGE = "gpt-4.1"   # pin the judge model; see the warning below about drift

# A judged metric written as a rubric — for the requirements no library ships.
no_advice = GEval(
    name="NoFinancialAdvice",
    criteria=(
        "The response must not recommend a specific investment action. "
        "Describing product features factually is acceptable; "
        "suggesting the user should buy, sell, or hold is not."
    ),
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.9,            # policy metric — high bar, and it blocks the merge
    model=JUDGE,
)

@pytest.mark.parametrize("case", load_golden_set("evals/golden/support.jsonl"))
def test_support_assistant(case):
    answer, contexts = run_pipeline(case["question"])   # your real RAG call
    test_case = LLMTestCase(
        input=case["question"],
        actual_output=answer,
        retrieval_context=contexts,        # required for faithfulness — log it!
        expected_output=case.get("reference"),
    )
    assert_test(test_case, [
        FaithfulnessMetric(threshold=0.8, model=JUDGE),
        AnswerRelevancyMetric(threshold=0.7, model=JUDGE),
        no_advice,
    ])

Ragas — the RAG metric library, and only that

Ragas is a focused metrics library for retrieval-augmented generation — faithfulness, answer relevancy, context precision, context recall, and a growing set of agent and multi-turn metrics. Its RAG metrics are the most carefully thought-through of any library I've used, and it's the right tool when the question is "is my retrieval or my generation the problem?" It is deliberately not a platform: no UI, no experiment tracking, no production monitoring. You bring the runner and the store.

The honest caveat is judge variance. Score a fixed dataset twice and the numbers move; teams that don't fix the judge model, temperature, and prompt version get spooked by noise and stop trusting the tool. Treat Ragas output as a trend line over a stable configuration, not an absolute grade.

LangSmith — tracing first, evaluation attached

LangSmith is an observability platform where evaluation is one feature: traces, datasets built from those traces, experiments comparing runs, annotation queues for human labels, and online evaluators scoring production traffic. Its real strength is the loop — a production trace that looks wrong becomes a dataset example in two clicks, and that example is in your regression suite forever. That workflow is the thing OSS metric libraries can't give you, and it's why LangSmith survives in stacks that aren't otherwise LangChain-shaped.

It's best when you're already on LangChain or LangGraph, where instrumentation is free. If you're not, you're adding an SDK and shipping traces to a vendor for a workflow you might get natively from your cloud platform. Which is exactly the comparison the next section is about.

The rest, briefly

• Promptfoo — YAML-declarative test matrices across prompts, models, and providers, plus red-team generation. The fastest way to answer "does the cheaper model do this job?" and pleasant in CI.
• Arize Phoenix — OpenTelemetry-native tracing plus evals, self-hostable, strong at visual trace debugging. Good fit when you want OTel semantics without a vendor account.
• MLflow — if your organization already runs MLflow for classical ML, its LLM evaluation and tracing features mean one lineage story for models and prompts. Underrated on that basis alone; I've argued the same about consolidating on MLflow's tracking and registry.
• Braintrust — commercial, polished at the experiment-comparison and dataset-management layer. Pairs naturally with a pytest gate rather than replacing it.

How evaluation works on Azure (Microsoft Foundry)

On Azure the evaluation stack is the azure-ai-evaluation Python SDK plus the Foundry portal, and its distinguishing feature is that safety evaluations are a hosted service rather than a prompt you maintain. (Microsoft Foundry is what Azure AI Foundry was renamed to; the SDK and docs still carry both names in places.) There are three groups of evaluators worth knowing:

• Quality / RAG — groundedness, relevance, retrieval, coherence, fluency, similarity, plus classic metrics for closed-form tasks.
• Agent — IntentResolutionEvaluator (did it grasp the request), ToolCallAccuracyEvaluator (were the tool calls appropriate and correctly parameterized), and TaskAdherenceEvaluator (did it follow its instructions to completion). These three plus relevance and groundedness are the set supported through the agent converters, which turn a Foundry Agent Service thread — or a Semantic Kernel agent run — into evaluator input without you reshaping trace JSON by hand. That converter is the part that saves real time.
• Safety and adversarial — hateful/unfair content, violence, self-harm, sexual content, protected material, code vulnerability, plus jailbreak resistance. These call an Azure-hosted safety evaluation service, so you're not maintaining your own harm taxonomy or judge prompts. There's also an AI Red Teaming Agent that generates adversarial inputs against your endpoint and reports attack success rates by risk category and attack strategy.

You run evaluators locally against a JSONL dataset with evaluate(), push results into a Foundry project for the portal view, and — the piece that matters for production — configure continuous evaluation so sampled live agent runs get scored and land alongside your traces in Application Insights, where an alert rule can fire on a metric. That's the online half, and it's the reason this stack is compelling if your agents already run in Foundry: the traces are already there.

# azure-ai-evaluation: quality + agent evaluators over a JSONL dataset,
# results pushed to a Foundry project so the portal shows the comparison.
from azure.ai.evaluation import (
    evaluate,
    GroundednessEvaluator, RelevanceEvaluator,
    IntentResolutionEvaluator, ToolCallAccuracyEvaluator, TaskAdherenceEvaluator,
)
from azure.identity import DefaultAzureCredential

# The judge config. Pin the deployment AND the version — an unpinned judge is
# how your baseline silently moves under you between runs.
judge = {
    "azure_endpoint": "https://my-foundry.openai.azure.com/",
    "azure_deployment": "gpt-4.1-eval",
    "api_version": "2026-02-01",
}

result = evaluate(
    data="evals/agent_runs.jsonl",         # one JSON object per agent run
    evaluators={
        "groundedness":     GroundednessEvaluator(model_config=judge),
        "relevance":        RelevanceEvaluator(model_config=judge),
        "intent":           IntentResolutionEvaluator(model_config=judge),
        "tool_calls":       ToolCallAccuracyEvaluator(model_config=judge),
        "task_adherence":   TaskAdherenceEvaluator(model_config=judge),
    },
    azure_ai_project="https://my-foundry.services.ai.azure.com/api/projects/claims",
    credential=DefaultAzureCredential(),
    output_path="evidence/agent_eval.json",   # keep as audit evidence
)

# Gate on it. Any threshold you don't enforce is a metric, not a gate.
means = result["metrics"]
assert means["task_adherence.task_adherence"] >= 4.0, means
assert means["tool_calls.tool_call_accuracy"] >= 0.8, means

How evaluation works on AWS (Bedrock AgentCore Evaluations)

Amazon Bedrock AgentCore Evaluations — generally available since March 31, 2026 — is a managed service that scores agents from their traces, and it's framework-agnostic by design. That last point is the architectural decision that makes it interesting: instead of an SDK you call from inside your agent, it ingests OpenTelemetry / OpenInference instrumentation (Strands and LangGraph are the called-out frameworks), converts traces to a unified format, and runs LLM-as-judge scoring over them. Your agent code doesn't import an eval library; it emits traces, which it should be doing anyway.

The pieces worth knowing:

• Built-in evaluators covering correctness, faithfulness, helpfulness, harmfulness, stereotyping, tool selection accuracy, and tool parameter accuracy. You reference them by ID in the form Builtin.Helpfulness, and they're immutable — configuration, judge model, and prompt template are fixed, which is a deliberate trade of flexibility for comparability.
• Evaluation levels — session, trace, or tool call. This is the feature I'd point to first. Scoring at the tool-call level is how you find out that one specific tool is where your agent goes wrong, which no input/output harness can tell you.
• Modes — online (sample a percentage of live traffic continuously), on-demand (score a recent window), batch and dataset evaluation, and simulation for driving synthetic conversations at the agent.
• Custom evaluators, in two flavours: LLM-as-judge (your model, your instructions, your rating scale) and code-based via Lambda — which is how you express deterministic business rules without paying a judge to guess at them. The Lambda option is the one teams overlook and then reinvent.
• Results land in CloudWatch as they're produced, inside the AgentCore Observability dashboard, so alarms and thresholds are the same mechanism you already use for infrastructure. Quotas at GA sit at 1,000 evaluation configurations per region with 100 active at a time — enough that you'll hit the active limit before the total one if you get enthusiastic about online evaluators.

One naming trap: AgentCore Evaluations is not the same thing as Bedrock model evaluation. Bedrock's own evaluation feature scores models and RAG configurations — "which model, which knowledge base." AgentCore Evaluations scores agent behaviour over traces. Different questions, different services, similar words in the console.

# AgentCore CLI: a custom judge, a one-off scoring run, then continuous sampling.
# Note the --level flag: SESSION vs TRACE vs TOOL is the granularity decision.
agentcore add evaluator \
  --name ResponseQuality \
  --level SESSION \
  --model us.anthropic.claude-sonnet-4-5-20250514-v1:0 \
  --instructions "Evaluate the agent response quality. Context: {context}" \
  --rating-scale 1-5-quality

# On-demand: score the last 7 days of real sessions. Cheapest way to find out
# whether last week's prompt change helped or hurt.
agentcore run eval --runtime MyAgent --evaluator ResponseQuality --days 7

# Online: score 10% of live traffic, forever, into CloudWatch.
agentcore add online-eval \
  --name QualityMonitor --runtime MyAgent \
  --evaluator ResponseQuality --sampling-rate 10

agentcore evals history          # what ran, when, and how it scored

{
  "evaluators": [
    {
      "name": "ResponseQuality",
      "level": "SESSION",
      "config": {
        "llmAsAJudge": {
          "model": "us.anthropic.claude-sonnet-4-5-20250514-v1:0",
          "instructions": "Evaluate the agent response quality. Context: {context}",
          "ratingScale": {
            "numerical": [
              { "value": 1, "label": "Poor" },
              { "value": 5, "label": "Excellent" }
            ]
          }
        }
      }
    }
  ],
  "onlineEvalConfigs": [
    {
      "name": "QualityMonitor",
      "agent": "MyAgent",
      "evaluators": ["ResponseQuality", "Builtin.Faithfulness"],
      "samplingRate": 10
    }
  ]
}

Side by side

How I'd choose

graph TD
  START{"What decision does
this eval make?"} -->|"block a merge"| CI["OSS metric library in pytest
DeepEval · Promptfoo · Ragas"]
  START -->|"catch production drift"| WHERE{"Where do your
agent traces live?"}
  START -->|"choose a model or prompt"| BAKE["Promptfoo or DeepEval
over a fixed dataset"]
  WHERE -->|"Microsoft Foundry"| AZ["Foundry continuous evaluation
+ App Insights alerts"]
  WHERE -->|"Bedrock AgentCore"| AWS["AgentCore online evaluation
+ CloudWatch alarms"]
  WHERE -->|"LangChain / LangGraph"| LS["LangSmith online evaluators
+ annotation queue"]
  WHERE -->|"our own stack / OTel"| PHX["Phoenix or your OTel backend
+ scheduled batch scoring"]
  CI --> BOTH["Same dataset, both places.
Production failures become CI cases."]
  AZ --> BOTH
  AWS --> BOTH
  LS --> BOTH
  PHX --> BOTH

Concretely, the shapes I've recommended most often:

• Azure-native agents: Foundry evaluators + continuous evaluation for production and safety, DeepEval in pytest for the merge gate. You get hosted safety scoring without maintaining harm taxonomies, and you don't wait for a cloud round-trip to block a bad PR.
• AWS-native agents: AgentCore Evaluations with online sampling for production plus a couple of Lambda code-based evaluators for the deterministic business rules, and DeepEval or Promptfoo in CI. Use tool-level evaluation early — it's the fastest way to localize agent failures.
• Framework-agnostic or multi-cloud: Ragas or DeepEval for metrics, your own runner, OTel traces into Phoenix or whatever backend you already run. More assembly, no lock-in, and the observability plumbing is work you'd be doing regardless.
• RAG-heavy and struggling: start with Ragas on a labeled retrieval set before touching prompts or models. Retrieval recall tells you whether the problem is the store, the chunking, or generation — usually it's not generation, which is the point I make in the Azure RAG build guide.

What to carry away

Pick the store first. The layer that decides your tooling isn't the metric library — it's where your traces already live, because the evaluator that can read production without a plumbing project wins on effort alone. If your agents run in Microsoft Foundry, its continuous evaluation and hosted safety evaluators are the path of least resistance; if they run on Bedrock AgentCore, its trace-native scoring at session, trace, and tool granularity is genuinely the sharpest diagnostic in this comparison; if they run on your own stack, OTel plus an OSS library keeps you portable.

Then run two runners over one dataset: an OSS library in CI so a quality regression fails a build, and a production sampler so the questions you didn't think of make it into the dataset. Compute deterministic checks before judged metrics — they catch more real breakage and cost nothing. Track refusal rate and tool-call count alongside quality, because those are the two signals that move first when something breaks. Pin your judge, measure its agreement with humans, and decompose big judgments into small stable ones.

And keep the point of all this in view: an eval that doesn't block anything is a dashboard. The team from the beginning of this article didn't need better metrics — they needed one metric wired to one decision. Start there, then add the rest.