Guide
Last updated: 26 September 2026
Evaluation is what turns 'the agent feels better' into a number a release can be gated on. These are the steps, in order — and this is the method the Zenovae audit uses.
Evaluate an agent in four steps: define the task, sample traces, score the final answer and the tool choice, and set a release gate. This is the method the Zenovae Agent Reliability Audit applies over 10 business days.
Write the task the agent is supposed to complete, the success condition, and the failures that block release. For a live customer support agent, 'answers support questions' is not a task. 'Resolves tier-1 password, order-status, and returns-initiation requests without a human touch, and hands off everything else' is.
The failure list matters as much as the success condition. Hallucinated policy, wrong refund promised, account action on the wrong customer — these are the failures you refuse to ship, written down before any scoring.
Pull real or staging runs. Not the curated demo inputs — a sample wide enough to include the ugly middle: ambiguous requests, multi-intent messages, the ones where the tool timed out. If no traces exist, producing them is the first two days of work, and nothing else can start until they do.
Did the agent complete the task, end to end? A correct answer that did not create the return label is a failure, not a partial credit.
Did it call the right tool, with the right arguments, in the right order? Wrong-tool calls are the failure mode that survives a good-looking final answer.
Total spend divided by completed tasks — not per run. Retries and hallucination-recovery loops hide in the gap between those two numbers.
Every irreversible action without an approval rule is a gap. Count them. Each one is a decision the business never made.
Turn the scores into a pass/fail condition before the next release: task success above a stated bar, zero blocked-failure types present, cost per successful task under budget. The gate is written down, so the launch conversation is about the number, not about enthusiasm.
This four-step method is exactly what the Agent Reliability Audit runs against a live or pre-launch agent, delivered as a written score and the three changes that would move it.
Define the tiers it may resolve and the failures it must never commit, sample real conversation traces, score task success and tool choice per run, compute cost per successful task, and gate release on written thresholds. The same method applies to any live agent.
Observability collects what happened. Evaluation judges whether it was right. Dashboards without scoring give you visible failures; scoring gives you a number you can gate a release on.
Enough to include the ambiguous and failing middle, not just the happy path. Start with a sample wide enough that each blocked-failure type has a chance to appear, then widen where the score is uncertain.
Someone who did not build the agent and can say no to a release. That is either an internal owner with a written gate, or an external audit — the Agent Reliability Audit exists for teams that do not have the first.
The Agent Reliability Audit applies this method to your agent over 10 business days: a written score and the three changes that would move it.
Start a scoping call