The agent was working. That was the problem. Six weeks after we shipped a support-triage agent, the platform lead sent me a screenshot of the Azure bill with one line highlighted and a single question: "is this right?" It was right. The per-call numbers on our dashboard were also right — a median of about four tenths of a cent per model call, which had looked delightful in the design review. Nobody had multiplied it by the number of calls a real agent makes to finish a real job.
That multiplication is the whole subject of this article. A chat completion has one call per answer, so cost per call and cost per outcome are the same number and you can be sloppy. An agent has a loop, and the loop's length is decided at runtime by a model — which means your unit cost is a distribution you don't control, not a constant you can quote. I've written about the platform-level view of this in LLM and AI FinOps: caching, routing, output-token asymmetry, and the visibility-attribution-optimization loop. This piece is narrower and more operational — the metric I now insist on before an agent goes to production, how to instrument it, and the five levers that actually move it.
The only unit that matters
Cost per resolved task: total spend divided by the number of tasks the agent finished acceptably, over the same window. Not per call, not per token, not per conversation. The denominator is doing the heavy lifting — it forces you to define what "resolved" means and to count the failures, which is precisely what per-call metrics let you avoid.
Two agents can have identical per-call costs and unit economics that differ by an order of magnitude, because the numbers that separate them are the ones nobody puts on the dashboard:
| Factor | Why it dominates | Where it hides |
|---|---|---|
| Steps per task | Linear multiplier on everything else. Nine steps instead of three is 3× the bill for the same outcome. | Averaged away — the mean is 3.4, the p95 is 14, and the p95 is most of your spend |
| Context re-sent per step | Each step re-sends the growing history, so cost grows quadratically with steps, not linearly | Invisible unless you log prompt tokens per step, not per task |
| Resolution rate | A task that fails still cost full price and then a human does it anyway — you pay twice | Nobody joins spend to outcome, so failures are free in the reporting and expensive in reality |
| Retry and loop behaviour | A tool that returns an ambiguous error can send the model round again, indefinitely | Looks like "the agent is thinking," reads on the bill like a denial-of-wallet attack you ran on yourself |
| Human minutes saved | The numerator of the business case; without it the cost number has no scale | Owned by ops, not engineering, so it never lands in the same document |
The quadratic point deserves a moment because it surprises people who reason about cost per token. In a naive loop, step n re-sends everything from steps 1 to n−1. Twelve steps of a conversation that grows by a page each time is not twelve pages of input — it is closer to seventy. That's why a small change in average step count moves the bill so much more than a switch to a cheaper model does.
graph TD
TASK["Task arrives"] --> LOOP{"Agent loop
step n"}
LOOP -->|"prompt = system + tools + history(1..n-1)"| MODEL["Model call
input tokens grow every step"]
MODEL -->|"tool call"| TOOL["Tool executes
result appended to history"]
TOOL --> LOOP
MODEL -->|"final answer"| DONE{"Resolved?"}
DONE -->|"yes"| WIN["cost / resolved task"]
DONE -->|"no — human takes over"| LOSS["cost paid twice:
tokens + human minutes"]
LOOP -->|"step cap hit"| CAP["Escalate deliberately
bounded loss"]
The two exits that decide your economics. The resolved path is the one your business case assumed; the human takes over path is the one that quietly doubles unit cost, because you paid for the attempt and then paid for the person. The step cap is the third exit, and the only one you control — it converts an unbounded loss into a bounded one and hands the task to a human early rather than expensively.
Instrumenting it, concretely
You need three things on every agent run, and if you only get one, make it the first: a task id that survives the whole run, token counts per step (input and output separately), and a resolution verdict. Traces give you the first two almost free if you're emitting OpenTelemetry — the point I keep making about what to instrument before your first incident. The verdict is the part teams skip, and without it you have spend data and no denominator.
# The minimum honest ledger. One row per step, one verdict per task — that's it.
# Everything else (dashboards, alerts, per-tenant chargeback) is a query away.
from dataclasses import dataclass, field
PRICE = { # $ per 1M tokens; pin these, they change
"gpt-4o": {"in": 2.50, "out": 10.00},
"gpt-4o-mini": {"in": 0.15, "out": 0.60},
}
@dataclass
class TaskLedger:
task_id: str
tenant: str
steps: list = field(default_factory=list)
def record(self, model, in_tok, out_tok, tool=None, ms=0):
p = PRICE[model]
self.steps.append({
"model": model, "in": in_tok, "out": out_tok, "tool": tool, "ms": ms,
"usd": (in_tok * p["in"] + out_tok * p["out"]) / 1_000_000,
})
def close(self, resolved: bool, escalated_to_human: bool):
cost = sum(s["usd"] for s in self.steps)
emit_metric("agent.task.usd", cost, tenant=self.tenant, resolved=resolved)
emit_metric("agent.task.steps", len(self.steps), tenant=self.tenant)
# The join that makes the number mean something: spend AND outcome,
# on the same record, so cost-per-RESOLVED-task is a division not a guess.
emit_metric("agent.task.resolved", 1 if resolved else 0, tenant=self.tenant)
if escalated_to_human:
emit_metric("agent.task.human_handoff", 1, tenant=self.tenant)
return costThen the number you report is a single division over a window: sum(agent.task.usd) / count(resolved == true). Report it next to the p95 of agent.task.steps, because those two together tell you whether your problem is the model, the loop, or the task mix. And segment by tenant from day one — in every deployment I've measured, a small number of tenants or task types generate a wildly disproportionate share of spend, and you cannot see that in an average.
The five levers, in order of leverage
1. Cap the steps, and escalate on the cap
The cheapest change with the largest effect. A hard maximum on tool calls per task turns your worst case from unbounded into arithmetic, and the escalation path it forces is one you wanted anyway. Set the cap from your own p95, not from a round number — if 95% of successful tasks finish in six steps, a cap of ten is generous and a cap of forty is a budget with no owner. Do the same for wall-clock time and for total tokens per task; whichever trips first, stop and hand off.
2. Stop re-sending what the model doesn't need
The quadratic term is attackable. Summarize or drop tool output that has served its purpose, keep a compact running state instead of a verbatim transcript, and don't paste an entire document into the history when a reference and a retrieval will do. In practice this is the difference between a twelve-step task costing seventy pages of input and costing twenty. It also improves quality, because a shorter, cleaner context is easier for the model to reason over than a long one full of stale tool dumps — a point that ties directly to how you design the agent's state in the first place.
3. Route by step, not by agent
Most teams pick one model per agent. Most agent loops contain steps of wildly different difficulty: deciding which tool to call is often trivial, while synthesizing a final answer from six sources is not. Routing the easy steps to a small model and reserving the frontier model for the hard ones is the standard move, and the mechanism belongs in the gateway rather than in each agent's code. Two cautions from experience: measure quality per step class before and after, because a cheaper router that adds two steps has cost you money; and keep the fallback path priced, since an outage that silently promotes everything to the expensive model is a bill surprise.
4. Make tools return less, and return it unambiguously
An underrated lever, because tool output is input tokens on every subsequent step. A tool that returns a 400-row JSON blob when the model needs three fields is charging you rent for that blob at every step after it. Trim tool responses to what the model can act on, paginate deliberately, and — this is the part that pays twice — make error responses actionable. An error the model can't interpret produces a retry; a retry is a full extra step; a few of those per task is a material fraction of your unit cost. "Not found: no customer with id X, try search_customer" ends the loop. "Error: 400" restarts it.
5. Cache what repeats, and know which cache you're using
Prompt caching pays for the stable prefix of your context — the system prompt and tool schemas — which in an agent is re-sent on every step, so the discount compounds across the loop rather than applying once. That's a structural argument for keeping the stable part of your prompt genuinely stable, and for ordering the context so the volatile parts sit at the end. It's also worth being precise that this is provider-side prefix caching, not a semantic response cache; they solve different problems and I've seen the two conflated in a design review with an unhappy ending.
The failure that only shows up when the agent is working. A model upgrade improved our tool selection and doubled our bill in the same week. The agent had started attempting harder tasks it would previously have bounced early, which is exactly what we wanted — more resolutions — but the mean step count moved from three to five and the p95 from nine to twenty-two, and nobody was watching that metric because quality had improved. Two lessons stuck. First: track steps-per-task and cost-per-resolved-task on the same dashboard as your quality metrics, because an improvement on one axis routinely moves the other and you want to see the trade rather than discover it on an invoice. Second: alert on the p95, not the mean. The mean is dominated by easy tasks and moves slowly; the tail is where a loop that has started to thrash announces itself first. Add a per-tenant daily budget with a hard stop, too — the same defence you'd want against an adversarial prompt trying to make your agent spend money, and cheap enough that there's no reason not to have it.
Putting a number on the business case
The reason this metric matters commercially is that it makes the comparison an executive actually needs: cost per resolved task versus cost per task done the old way. If a human handles a triage in eleven minutes at a loaded cost you can look up, you have a ceiling. Your agent's number has to sit under it with room for the escalation rate — because unresolved tasks cost the agent's tokens plus the human's minutes, so a low resolution rate can put you above the ceiling even with a cheap loop.
The arithmetic is unforgiving and clarifying. An agent resolving 60% of tasks at a modest per-task cost can be worse than no agent at all once you add the handoff overhead and the review time for the ones it got wrong quietly. An agent resolving 85% at twice the token cost is usually a clear win. That's why I'd rather spend the optimization effort on resolution rate than on shaving tokens — the denominator moves the ratio more than the numerator does.
graph LR A["Instrument
task id · tokens/step · verdict"] --> B["Measure
cost per resolved task
+ p95 steps"] B --> C{"Above the
human-cost ceiling?"} C -->|"yes, and resolution is low"| D["Fix resolution first
better tools, retrieval, prompts"] C -->|"yes, and resolution is fine"| E["Fix the loop
step cap · context · routing"] C -->|"no"| F["Ship, then watch the p95
and the per-tenant tail"] D --> B E --> B
The order of operations that keeps you from optimizing the wrong thing. Cheap tokens on an agent that fails half its tasks is a cheaper way to not solve the problem. Fix the denominator, then the numerator, then keep watching the tail — which is where regressions arrive first.
What to carry away
Per-call cost is a number that makes agents look free; cost per resolved task is the number that tells you whether to ship one. Instrument three things — a task id that spans the run, per-step input and output tokens, and a resolution verdict on the same record as the spend — and the metric becomes one division instead of a quarterly argument.
Then work the levers in order: cap the steps and escalate on the cap, so your worst case is arithmetic rather than a surprise; stop re-sending context the model doesn't need, because the loop makes that cost quadratic; route per step rather than per agent, in the gateway; make tools return less and fail informatively, since an uninterpretable error buys you a whole extra step; and lean on prefix caching by keeping the stable part of your prompt actually stable. Watch the p95 of steps-per-task next to your quality metrics, because the two move together and usually in opposite directions.
And keep the comparison honest. The agent has to beat the loaded cost of the human doing the same task, including the ones it hands back. When it does, the case makes itself; when it doesn't, no amount of token shaving will close the gap — the resolution rate will.