Model charges are only one part of the cost
An assisted test consumes calls, tokens, infrastructure, and wait time; that is measurable and depends on the provider, model, length, cache and retries. But a flow also costs preparing requirements and data, integrating permissions and environments, reviewing findings, maintaining prompts and adapters, investigating false positives, and responding to non-deterministic output.
That is why “the context is the real cost” is a useful hypothesis to guide an audit, not a proven economic law for every project. If the test context is messy, the model can fill in gaps with assumptions. If everything is sent indiscriminately, consumption, latency and noise increase. The goal is sufficient, versioned, and relevant context for a specific decision.
At a minimum, you need to know what you want to test and what the risk is; what version of the application is used; what should be prepared; what data you can use; what steps to follow; what is the expected result; what tools you have available; and what security and execution limits it must respect. You also need to define when it should stop without giving a verdict, for example, if credentials are missing, the environment fails, data conflicts, or it is not clear what counts as success.
Saves a snapshot of instructions, tool version, model, parameters, redacted input data, traces and result. This way you can replay a regression, compare versions, and distinguish a product change from an agent change. Keep secrets out of prompts and apply least privileges to tools and browsers.
How to tell whether an evaluation is worth the cost
Estimate the cost of each useful test and each confirmed defect. Include model and infrastructure charges, the time needed to prepare information, triage failures, review results, maintain the configuration, retry runs, and resolve integration problems. Also track how many alerts are correct or false, which risks are covered, whether a test can be repeated, how long the team waits for a result, and how many defects reach production. Generating more tests is not helpful if the team has to discard most of them.
Compare with the current alternative on representative tasks and over several iterations. METR found in 2025 that AI tools increased completion time by 19% in a small sample of experienced developers working on their own repositories. It was not a study of testers nor does it prove that tools always delay: it teaches that perceived and observed savings can diverge and that the work context changes the result.
The information given to a testing tool should not be a folder full of documents. It should include the current requirement, relevant rules, permitted states and data, expected results, execution limits, and the format for recording evidence. Record where each item came from and when it was last updated. If two documents conflict, flag and resolve the disagreement before asking an AI system to interpret it.
Keep the material concise and current: the active requirement, relevant rules, permitted states and data, expected result, execution limits, and evidence format. Record the source and update date for each item. If two documents conflict, flag the disagreement and resolve it before asking an AI system to interpret the requirement.
Balance coverage, useful signals, and review effort
More context does not guarantee a better answer. A prompt that includes outdated requirements can worsen the result; one that is too short may omit preconditions. Test the package with known tasks and evaluate whether the agent detects expected risks, avoids duplicates, recognizes when it doesn't know, and delivers reproducible evidence. Maintain a small regression set to compare context and model changes.
Also measures subsequent work: minutes until a finding is confirmed, percentage of alerts accepted, repetition of the same false positive, and failures that appear after deployment. A less abundant but reviewable signal can be worth more than hundreds of alerts. Optimize time to a reliable decision, not tokens saved or cases generated.
“Context is the real cost” is a useful provocation, not a proven economic rule for every project. In a test that makes only a few model calls, usage fees may be the largest expense. In a workflow that also requires permissions, environment setup, result review, and data maintenance, people’s time and operating costs may matter more. The balance depends on the model and modality, current prices, input and output size, caching, retries, and review effort. Comparing only the price per million tokens—or the cost of one example call—leaves out much of the picture.
Providers publish different price lists and update them over time. Some features cost extra; free tiers may have usage limits or data conditions. Use the current price and contract for the service you would actually run, and record when you made the estimate. Include billed input and output, any billed reasoning, external tools, and storage. Published rates help estimate a budget; they are not the total cost of a workflow or a promise of savings. Compare two ways of doing the same work against the same quality and safety criteria.
Calculate the cost of a useful decision
Calculate cost per run that ends in a reviewable decision, not per model call. Include token and tool charges; time spent gathering and clarifying requirements; preparing data; setting up access and test environments; reviewing results; investigating alerts; correcting context; maintaining integrations; and rerunning tests. Track work that would have happened anyway separately so it is not counted twice. If you use an hourly rate, state the assumption and include both direct cost and opportunity cost: a tester may need to pause another task to verify a finding.
Then calculate at least three measures: cost per accepted run, per confirmed defect, and per risk covered. Each tells only part of the story. A safety check can be valuable even when it finds no bug; one confirmed critical defect may deserve more review than a hundred routine cases. Include the false-positive rate and the time people spend classifying alerts. A cheap model that produces alerts nobody trusts may cost more than a pricier run that yields reusable evidence—but measure that in the actual workflow.
A useful context package starts with the objective and risk, the version being tested, the acceptance criteria, the preconditions, the allowed data, and the evidence format. Add contracts, states, and rules only when they change the decision of that test. Remove secrets, obsolete documents and unrelated material. If two sources contradict each other, record the conflict; no esperes que el modelo adivine cuál tiene autoridad. Each fragment must be traceable to a current requirement or decision.
Versiona ese paquete junto con el caso y registra qué modelo y herramientas lo recibieron. A small regression test can compare context changes: expected cases should still be detected, ambiguous ones should produce a question or abstention, and outputs should include sufficient evidence. This is how you discover if removing context reduces costs without introducing omissions, or if adding it only increases the input. The goal is not to give the agent all the project documents; is to give you exactly what you need for a controlled check and repeat it when you change the product.
Where rework tends to hide
Rework does not appear only when the model makes a mistake. It also appears when an integration requires adapting the format of the results, when insufficient permissions interrupt execution, or when an update changes the behavior and requires revision of prompts and assertions. Count those cases separately. If everything is added to “AI time”, no one will know whether to change the model, improve the API contract, stabilize the environment or reduce duplicate reviews. On a simple sheet, each repetition can indicate stage, cause, time spent, and whether it produced new evidence. After several weeks, a few reasons usually appear that explain much of the cost.
The economic comparison must use a task that the team really needs to perform. For example, measure the review of medium-risk changes over several iterations, with the same acceptance rules and a comparable group. It counts from preparation to final decision, includes failed executions and notes cases where a person had to redo the work. It presents median and range in addition to the average, because a couple of extreme cases can distort it. The result will not be a universal fare for AI QA; will be a useful estimate for that task, with assumptions that someone else can review and update.
Finally, repeat the calculation when the usage volume, model, rates or coverage change. A small controlled trial may seem cheap because it still depends on manual work that is not accounted for; a stable operation can spread the integration cost over many executions. Write down these conditions and present low, medium, and high usage scenarios instead of a single figure with false precision. With that discipline, the team can decide what to automate, what to keep manual, and when to re-evaluate the investment.