How to evaluate an AI agent before it touches customers

Practice · reviewed August 2026

You evaluate an AI agent by fixing a set of real inputs with known-correct outputs, running the agent against that set on every change, and measuring how often it is right, how often it is wrong, and how often it correctly refuses. Without that set, every prompt change is a guess and every regression is invisible until a customer finds it. Twenty real cases assembled in an afternoon is the difference between engineering and hoping.

Build the evaluation set first

Take fifty real examples from the last month of actual work, deliberately including the ones that went wrong, the ambiguous ones, and the ones where the right answer was to escalate to a person. Record what a competent human did in each case. That is your set.

It does not need tooling to start. A spreadsheet of inputs and expected outputs, run by a script, catches most regressions long before a platform would.

Measure four things, not one

• Correct: the agent did the right thing. The obvious one, and the least informative on its own.
• Wrong: it acted, and was incorrect. The number that decides whether it can be trusted with anything consequential.
• Refused correctly: it recognised it could not complete the task and escalated. This is a success, and counting it as a failure trains you to build something dangerous.
• Failed silently: it produced plausible output without doing the work. The one that matters most, and the only one that requires reading the logs rather than the outputs.

What to log while it runs

Every tool call with its arguments and result, the plan the agent formed, the point at which it deviated, and the token cost of the run. The most useful production alert is rarely an error rate — it is a sudden drop in tool calls, which usually means the agent stopped doing the work and started improvising an answer.

What an evaluation row actually looks like

Concretely, for an invoice-chasing agent, one row is: the input is a forwarded email thread with the invoice attached as a scan; the expected outcome is that the agent identifies invoice 4417, finds it is nineteen days past terms, drafts a chase to the billing contact and stops for approval; the failure to catch is the agent inventing an invoice number because the scan was unreadable.

That last column is the one most teams leave out, and it is the one that matters. Writing down how each case is likely to fail turns an evaluation set from a scorecard into a design document — you usually discover the guardrail you need while writing it, before any code exists.

The gate before production

An agent should not touch a customer until: it passes the evaluation set at an agreed threshold, its wrong-action rate is measured rather than assumed, everything consequential sits behind a human approval enforced in code, someone is named as owner, and there is a documented fallback to the manual process while it is down.

None of that requires a platform. All of it requires deciding it matters before launch rather than after the first incident.

The eight ways this goes wrong

Common questions