Outcome vs process metrics for agents

Ask most teams how their agent is doing and you get one number. It is a start, but it cannot tell you whether a regression came from retrieval, reasoning, or tool use, and it says nothing about whether the agent got the right answer efficiently or by burning ten thousand tokens and five tool calls it did not need. A useful evaluation has two layers: outcome metrics that measure whether the agent succeeded, and process metrics that measure how it got there and let you localize failures.

Outcome metrics: did it succeed

• Task completion / pass@1: the fraction of tasks the agent resolves correctly on the first try. For coding agents this often means a hidden test suite passes. This is your headline number.
• Resolve rate with retries: completion allowing k attempts. The gap between pass@1 and pass@k tells you how much reliability you are leaving on the table versus how much is genuinely out of reach.
• Human acceptance: for open-ended output, the rate at which a reviewer accepts the result unedited. Slower to collect, but the closest proxy for real value.

Process metrics: how it got there

Process metrics are what turn it got worse into retrieval stopped fetching the right file. The important ones:

• Context recall: of the code or facts the task needed, how much did retrieval actually put in context. Low recall means the agent was set up to fail before it reasoned at all.
• Context precision: of what retrieval fetched, how much was relevant. Low precision wastes budget and buries signal in noise.
• Tool-call validity: the fraction of tool calls that were well-formed and succeeded. A dip here points at prompt or schema drift, not reasoning.
• Step efficiency: turns and tokens per completed task. Two agents at the same completion rate are not equal if one uses half the turns.

Outcome metrics tell you the agent regressed. Process metrics tell you which component to open. You need both, or every regression becomes a bisecting expedition.

Weight quality by cost

In production, quality is never free: every point of completion has a token and latency price. Report quality and cost together, not as separate dashboards, and evaluate changes on the Pareto frontier: a change is unambiguously good only if it improves quality without raising cost, or lowers cost without hurting quality. Everything else is a tradeoff to be made deliberately. A single cost-weighted quality score, completion per dollar or per second, is a good headline for comparing configurations, as long as you keep the raw axes visible so you can see which lever moved.

Build the eval set before you optimize

None of this works without a fixed set of tasks to measure against. Assemble a golden set of representative tasks with known-good outcomes, sized large enough that a real change moves the number past noise. Treat it as a regression suite: run it on every meaningful change, and add any production failure you fix as a new case so it cannot silently come back. Keep offline eval, fast and deterministic, separate from online eval, real traffic and slower signal, which catches what your golden set missed.

The payoff

With outcome and process metrics on a fixed eval set, optimization stops being guesswork. You can route models, compress context, and rework retrieval and see immediately, on the same tasks, whether completion held, which component moved, and what it cost. That measurement loop is the difference between optimizing an agent and merely changing it.