A multi-step agent needs a budget for both resources and waiting time. Without one, a difficult task can trigger repeated model calls, long tool chains and retries that cost more than the outcome is worth. A budget makes the workflow’s stopping behavior part of the product design.
This article proposes a planning method rather than current vendor pricing. Cost and latency must be measured under the actual model, tools, workload and service conditions. A fixed estimate based on a single successful demonstration is not enough.
Budget the complete task
Include planning calls, retrieval, tool execution, validation, retries and any human review needed to deliver an acceptable result. Define the customer’s useful unit of work first. The AI feature unit-economics guide explains why cost per accepted task is more informative than cost per response.
Use separate limits for input size, number of steps, elapsed time, tool calls and monetary exposure where the infrastructure supports it. These limits address different failure patterns. A short but expensive external tool call can exceed a cost budget before a time limit is reached.
Allocate room for verification
| Budget component | Design question |
|---|---|
| Evidence collection | How much information is sufficient for this task? |
| Reasoning or generation | Which stages actually need model interpretation? |
| Tool execution | Which calls are necessary and can be bounded? |
| Verification | What must be checked before completion is claimed? |
| Recovery | How many retries are useful for each failure type? |
| Human review | When does waiting change the customer promise? |
Do not spend the entire budget generating a result and leave no capacity to verify it. A workflow that runs out of time before checking an external effect should report an unresolved state rather than a confident success.
Plan for the tail
Average latency can look acceptable while a minority of tasks take much longer. Examine distributions by task type and complexity. Long documents, unavailable websites and ambiguous tool responses may create a different cost profile from routine work.
Decide what the customer experiences near the limit: a partial result, a request for narrower scope, a queued continuation or a clear stop. The fallback should be designed in advance. Silently switching to a lower-quality path without communicating a material limitation can make the output misleading.
Treat retries as a bounded policy
Different errors deserve different responses. A temporary service problem may justify another attempt; invalid input or denied permission usually needs correction or escalation. Preserve the identity of the logical operation so a retry does not duplicate an external effect.
Stop conditions should exist outside the model’s own willingness to continue. OWASP identifies uncontrolled inference consumption as a source of resource exhaustion and economic loss. Its unbounded-consumption guidance supports enforcing limits at the application and service boundary.
Optimize the expensive step that matters
Measure before changing the architecture. Sometimes repeated retrieval dominates; sometimes a verification tool is slow; sometimes human correction costs more than all model calls. Reducing one visible latency number can worsen total task cost if it increases errors or review effort.
Compare a simpler workflow where possible. Cache only information whose freshness and access scope make reuse appropriate. Parallel work can reduce elapsed time for independent steps, but it can also increase cost and does not remove dependencies or authorization requirements.
Review budgets with quality outcomes
Track accepted tasks, timeouts, partial results, critical failures and actual resource use together. A cheaper system is not an improvement if it fails more often on the tasks customers depend on. A slower system is not automatically better if the extra steps add no verified value.
The replayable evaluation guide helps compare versions consistently. The observability guide provides the trace needed to explain overruns. A useful budget preserves enough capacity for a dependable result and makes the limits clear when the task cannot be completed within them.