Evaluating Agents

Evaluating a single model response is already awkward: the output is non-deterministic, correctness is often a judgement, and the thing you want to measure resists a simple assertion. All of that machinery is covered in evaluation harnesses for LLM systems and it is the prerequisite for this article rather than a substitute for it.

Agents are harder for a specific structural reason. The unit under test is not an output, it is a trajectory: a sequence of decisions, tool calls, results and revisions, ending in some state of the world. Two runs can produce an identical final answer by entirely different routes, one of which took four steps and one of which took thirty-one, tried a destructive operation twice, and got the right answer by luck. Grading only the answer scores those the same. They are not the same, and the difference is exactly what will hurt you in production.

There is also a failure mode that single-response evaluation has no vocabulary for. An agent that reaches the correct outcome after forty tool calls and a large bill has not succeeded. It has failed in a way that passes your test suite, and it will keep failing that way, quietly, at scale, until somebody looks at the invoice or the latency percentiles. Cost and step count are not operational telemetry to be reviewed later. They are assertions, and they belong in the harness next to correctness.

Build the trajectory dataset first

Everything below depends on having realistic tasks with known-good outcomes. Collect them from real usage rather than inventing them, because invented tasks are cleaner than real ones in exactly the ways that matter.

A usable case has: the initial state (including whatever the agent will read), the goal as a user would state it, the acceptable end states, and the constraints that must hold throughout. That last element is the one people omit and the one that catches the serious defects.

Stratify deliberately. A suite of thirty happy-path cases will be green forever and tell you nothing. You want the straightforward majority, the genuinely ambiguous, the ones where the right answer is to stop and ask, the ones where a tool fails midway, the ones where the request is outside scope and the correct behaviour is refusal, and the ones containing adversarial content. The last two categories are where teams are weakest and where incidents come from.

Start smaller than feels respectable. Thirty well-chosen cases that you actually run on every change beat four hundred that run monthly and whose failures nobody triages.

Outcome evaluation: necessary, insufficient

Outcome evaluation asks whether the end state is correct. Did the record get updated with the right values, did the answer contain the right facts, did the code compile and pass its tests, did the refund get issued for the right amount.

Where a deterministic check exists, use it and do not be clever. A schema check, a database assertion, a test suite result, a numeric comparison — these are cheap, stable and not subject to a grader's mood. Push as much of your evaluation into this category as you can; the effort spent making outcomes checkable pays back permanently.

Where the outcome is a judgement, a model-based grader against an explicit rubric is the practical option, with the usual caveats: the grader is itself a system that needs evaluating, it should be calibrated against human labels on a sample, and it should not be the same model doing the work if you can avoid it. Keep rubrics specific. "Was the response helpful" produces noise. "Does the response state the correct refund amount, name the policy applied, and avoid promising a timescale" produces a signal.

Report outcome success as a rate across the suite with the variance visible, and re-run on the same commit occasionally so you know what your own noise floor looks like. Without that figure you cannot tell a real regression from a bad afternoon.

Trajectory evaluation: asserting on the route

Trajectory evaluation asks whether the path was acceptable. It is where the agent-specific defects live, and it divides into invariants and quality signals.

Invariants are hard assertions that must hold on every run, and they should fail the build.

assert no call to a tool outside the allowed set for this task
assert no mutating call before the confirming read
assert every mutating call carried an idempotency key
assert no call to delete_* in any read-only scenario
assert the run terminated by its own stopping condition, not the cap
assert total steps <= 12 and total cost <= budget for this case

Note the last two. Terminating because it hit the iteration cap is a failure even when the answer happened to be right by then, because it means the agent did not know it was finished. And budgets belong here, as assertions, not in a dashboard nobody reads.

Quality signals are softer and are tracked as distributions rather than pass or fail. Steps per successful run. Redundant calls — the same tool with the same arguments more than once. Recovery rate after a tool error. How often the agent revisits a path it already abandoned. Tool selection accuracy where you can label the correct tool for a step.

Two composite measures are worth the trouble of computing. Efficiency ratio: actual steps divided by the minimum steps a competent operator would need for that case. A ratio near one is a well-designed tool set; a ratio of four means the agent is groping, and the cause is nearly always the tool layer rather than the model. Recovery rate: of the runs where a tool failed, how many still reached an acceptable outcome. That number predicts production robustness better than headline success does.

Replay harnesses with recorded tool responses

You cannot run a full evaluation suite against live systems on every commit. It is slow, it costs money, it mutates real state, and it is non-deterministic in a second dimension because the downstream systems change underneath you.

The answer is the same one distributed systems testing arrived at years ago: record and replay. Capture real tool interactions from live or staged runs — call, arguments, response, timing — and replay them in the harness. The model still makes fresh decisions; the environment is fixed.

This buys you a great deal. Runs become fast and free enough to execute on every pull request. The same fixed environment across runs isolates model and prompt changes from downstream noise. Failure cases that are hard to provoke — a timeout, a malformed payload, a rate limit, an empty result — become ordinary fixtures you can assert against. And a production incident becomes a permanent test case: capture the trajectory, add it to the suite, and it never recurs silently.

Two things to handle carefully. The agent may call a tool with arguments you never recorded, and your harness must decide deliberately: fail loudly, serve a nearest match with a warning, or fall through to a live call. Silent nearest-match is the option that quietly invalidates your results, so at minimum count and report the misses. And recordings go stale as the real systems evolve, which means a scheduled live run against a staging environment to detect drift, and a refresh policy with an owner. A replay suite that has diverged from reality is worse than none, because it is confidently green.

Regression suites in CI and catching silent degradation

Run the suite on every change to prompts, tool definitions, tool descriptions, model configuration, retrieval settings and orchestration code. All of those are production changes even though several of them look like content edits. A one-word change to a tool description can move behaviour more than a model upgrade, and if it is not gated by the same suite it will reach production ungated.

Gate on invariants, not on the success rate. Invariant violations fail the build absolutely. Success rate is noisy and should trigger on a statistically meaningful drop against a baseline computed over several runs, not on a single point below a threshold. Teams that gate hard on a noisy number learn to re-run until green, at which point the gate is theatre.

Silent degradation is the thing to design against, because it does not announce itself. Its signature is a headline number that stays flat while the underlying behaviour worsens. Watch these specifically.

Steps and cost per successful run creeping upward while success stays flat. This is the most common silent regression and the one that shows up on a bill rather than in a test.

Refusal or escalation rate drifting. Down means the agent has started attempting things it should hand off. Up means it has become useless in a way users will route around.

A single subgroup collapsing while the aggregate holds. Report per category, not just overall.

Recovery rate falling after a change to error messages or retry behaviour.

Tool selection shifting between two similar tools. Often harmless, occasionally the leading indicator of a large change in behaviour.

Keep a small set of canary cases running against production configuration on a schedule, and alert on their outcomes. Offline suites tell you what changed when you changed it; canaries tell you when the world changed underneath you, which is the case you did not initiate and therefore will not be watching for. The rest of that runtime picture belongs to running agents in production.

What the harness should report

One line per run, stored durably and queryable, is enough to answer nearly every question you will have.

Field	Why it earns its place
Case id and category	Lets you see subgroup collapse the aggregate hides
Outcome verdict and grader used	Separates deterministic checks from judged ones
Step count	The efficiency signal, and an early warning
Total cost	Turns an expensive success into a visible failure
Wall-clock duration	What the user experiences
Stop reason	Self-terminated, cap reached, error, escalation
Invariant violations	Hard failures, always gating
Tool call sequence	The raw material for every investigation
Config fingerprint	Model, prompt version, tool schema version

The last row is the one that gets added after the first incident where nobody could say which prompt version produced a result. Add it now. Comparing two runs is only meaningful if you know what differed between them, and in a system with this many moving parts you will not remember.

What to do on Monday

Take twenty real tasks from your logs, including at least five that went badly, and write down for each the initial state, the goal, the acceptable end states and the constraints that must hold. That is your first suite and it is worth more than any framework you could adopt this week.

Add three assertions to every case you already run: maximum steps, maximum cost, and terminated-by-own-stopping-condition. If your current pass rate drops when you add them, you have just learned that some of what you were recording as success was expensive failure.

Then start recording tool interactions from live runs so you can build a replay harness. Do this before you need it, because the useful recordings are the ones from incidents, and those only exist if capture was already switched on.

Finally, wire the suite to every change that touches prompts, tool descriptions, model configuration or orchestration, and gate the build on invariant violations only. Leave success rate as a reported trend with a baseline. A gate people trust is one they never re-run to get past.