Can a Coding Agent Batch Deterministic Tool Calls? A 60-Task Benchmark

Yohji Sakamoto — Tue, 21 Jul 2026 02:36:12 +0000

Disclosure: I maintain the open-source project Tura. This is not an independent review; I’m publishing the implementation and results so others can reproduce them and point out mistakes.

A long coding-agent task often repeats a familiar sequence: inspect the environment, update configuration, patch the bug, adjust tests, build, run tests and lint, then inspect Playwright screenshots. Not every transition requires a new model decision, but each tool call normally consumes another round.

Tura’s Macro layer batches deterministic steps into one tool call and returns structured results, including the exact failure point. If new evidence requires judgment, the macro stops and hands control back to the model; this is not blind batching.

On the same 60-task set we measured:

Configuration	Passes	Pass rate	Observed tokens	Rounds	Estimated cost
Macro + backward reasoning	48/60	80.0%	229,695,477	2,017	$221.138
Macro Direct	39/60	65.0%	75,108,167	969	$99.620
Codex CLI Medium	38/60	63.3%	333,538,349	3,140	$257.173
Codex CLI High	36/60	60.0%	455,742,296	6,074	$327.483

In this sample, Macro Direct achieved 1.7 percentage points more passes than Codex CLI Medium while using 77.5% fewer observed tokens and 69.1% fewer rounds. Against Codex CLI High, it used 83.5% fewer tokens and 84.0% fewer rounds, with a 5-point higher pass rate. Adding backward reasoning produced the highest pass rate, but also increased tokens and cost.

“Up to 84% fewer rounds” is not a universal guarantee. The useful denominator is total cost per completed, verified task, including failures, retries, and orchestration. These are aggregates from one 60-task set; other repositories can produce different ratios.

Code, scripts, tasks, and the reproduction method:

Where would you trust a deterministic macro in your agent workflow, and where should the model stop to inspect new evidence?

Why “90% Fewer Tokens” Can Still Mean a More Expensive Coding Agent

Yohji Sakamoto — Sun, 19 Jul 2026 15:41:25 +0000

Token-efficiency claims are everywhere in the coding-agent ecosystem: a plugin compresses shell output by 90%, a context tool removes thousands of tokens, or a prompt layer reports a large reduction in one step.

Those numbers may be real. The harder question is whether they reduce the cost of a verified, completed task.

I tested that question with a matched repository-rewrite task: rebuild the Rust eza project as a behavior-compatible Python implementation and satisfy 52 harness assertions. Every run used GPT-5.6-sol High through Codex CLI 0.144.1. I compared no plugin, Ponytail, and RTK, with two runs per condition.

What the matched runs showed

Condition	Mean verifier score	Mean tokens	Mean modeled cost	Mean rounds
No plugin	78.85%	6.660M	$5.281946	62.5
Ponytail	80.77%	-7.56% vs baseline	-8.87%	—
RTK	76.92%	+13.20% vs baseline	+7.18%	+44.00%

Ponytail used fewer tokens and cost less in this small sample, but took 13.51% longer. RTK used more tokens, cost more, and took 40.69% longer.

The important caveat is the sample size: n=2 per condition is not enough to estimate a causal plugin effect. In fact, the cost range divided by the mean was 43.25% without a plugin, 51.69% for Ponytail, and 30.78% for RTK. Those within-condition swings are much larger than the headline differences between conditions.

That means a single good run can easily produce a persuasive “saved X%” claim even when ordinary agent-trajectory variance is the simpler explanation.

The denominator is the real problem

The broader dataset contains 140 Codex CLI Medium/High runs, 10,365 agent rounds, 901,608,531 tokens, and $680.34 in modeled API cost. It contains no Tura runs.

The token and cost distributions were very different:

Cached input: 96.46% of tokens, 63.91% of modeled cost
New uncached input: 3.16% of tokens, 20.94% of cost
Model output: 0.38% of tokens, 15.14% of cost

So a tool can compress one prompt fragment or one class of terminal output by a huge percentage while barely moving the full-task bill. It may also change the agent’s path: extra searches, retries, tool calls, or rounds can erase the local saving.

For example, the dataset contains 1,082 uniquely classified RTK-supported shell calls returning 1,458,927 tokens—only 0.1618% of all task tokens. Even a perfect 90% reduction of every eligible return corresponds to about 0.96% in directly attributable modeled cost savings.

A better evaluation checklist

Before accepting a token-saving claim, I now ask four questions:

Was the same task repeated with and without the tool?
Did an external verifier confirm the final result?
Were retries, failures, orchestration, and tool calls included?
Is the metric cost per verified successful task, rather than tokens removed from one step?

Compression can still be useful. The point is that local compression percentage is not a reliable substitute for end-to-end evidence.

The full analysis, assumptions, formulas, and reproducible data links are here:

https://turaai.net/blog#token-saving-plugins-are-mostly-stupid-idea

The open-source project is here:

https://github.com/Tura-AI/tura

Disclosure: I maintain Tura and wrote the linked analysis. This post was prepared with AI assistance and reviewed against the underlying benchmark data before publication.

PromptZone - Leading AI Community for Prompt Engineering and AI Enthusiasts: Yohji Sakamoto

Can a Coding Agent Batch Deterministic Tool Calls? A 60-Task Benchmark

Why “90% Fewer Tokens” Can Still Mean a More Expensive Coding Agent

What the matched runs showed

The denominator is the real problem

A better evaluation checklist