A replayable operational task gives an agent a known starting state, a defined goal and a verifiable expected outcome. It lets a team compare versions against the same problem instead of relying on a collection of impressive demonstrations. For commerce workflows, the outcome must include external state and prohibited effects, not only the quality of the final explanation.
This article proposes an evaluation harness. It is not a report of a live agent’s success rate. All replay tasks should run against controlled fixtures or isolated test systems with no unintended customer impact.
Define the task contract
Record the user’s instruction, permitted resources, relevant business rules and completion condition. Include what must remain unchanged. For example, a task to identify orders needing review should not modify addresses, issue refunds or contact customers.
Use realistic but non-sensitive fixtures. The dataset should preserve the relationships that make the problem difficult without copying unnecessary personal information. Keep fixture versions stable so a changed result can be traced to the agent or rule version rather than an unnoticed data change.
Build a case portfolio
| Case family | What it tests |
|---|---|
| Routine completion | Basic task understanding and tool use |
| Missing information | Whether the agent asks or abstains appropriately |
| Conflicting evidence | Whether source authority is handled correctly |
| Partial prior execution | Whether completed effects are recognized |
| Temporary tool failure | Whether retries are bounded and safe |
| Unauthorized request in data | Whether the trust boundary holds |
| No action needed | Whether the agent avoids inventing work |
Include cases in which the correct result is escalation or no change. Otherwise the evaluation can reward unnecessary action and penalize appropriate restraint.
Score the final state and the path
Check whether the intended records were read or changed correctly, whether the output is grounded, and whether forbidden actions occurred. A final answer can be correct even after an unsafe intermediate action; that run should not pass unnoticed.
Measure task completion, critical violations, unsupported claims, unnecessary calls and human-review burden separately. Keep severe failures visible rather than averaging them away. The acceptable threshold depends on the workflow’s consequence and should be agreed before comparing versions.
Control nondeterminism without hiding it
Some agent behavior varies across runs. Repeat selected cases and report the distribution rather than choosing the best attempt. Where external services are mocked, make their responses and failure schedule explicit. Where a real test service is used, record its version and relevant state.
A reproducible environment does not require every generated sentence to be identical. It requires the important inputs and outcome checks to remain stable enough that differences can be interpreted. Equivalent correct explanations should not fail merely because their wording differs.
Compare against simpler workflows
A deterministic rule, a checklist-assisted operator or a single-step model call may solve the task adequately. Include a practical baseline. The agent’s additional steps should demonstrate value in quality, coverage or effort rather than being treated as sophistication by themselves.
Keep a held-out set that is not repeatedly used to tune prompts. Add new failures to a regression set while preserving unseen cases for broader evaluation. Otherwise the system can become highly adapted to the visible test collection without improving general behavior.
Connect evaluation to release decisions
For every release candidate, preserve the case results, version identifiers and unresolved failure categories. Define which failures block a release and which can be accepted only within a narrower pilot scope. A passing offline suite should lead to a bounded monitored deployment, not an unrestricted autonomy claim.
The observability guide explains how live traces can feed new replay cases. The website-reading threat model adds adversarial scenarios. A useful evaluation program accumulates evidence about what the agent can do reliably and makes its remaining limits difficult to overlook.