Token-efficiency claims are everywhere in the coding-agent ecosystem: a plugin compresses shell output by 90%, a context tool removes thousands of tokens, or a prompt layer reports a large reduction in one step.
Those numbers may be real. The harder question is whether they reduce the cost of a verified, completed task.
I tested that question with a matched repository-rewrite task: rebuild the Rust eza project as a behavior-compatible Python implementation and satisfy 52 harness assertions. Every run used GPT-5.6-sol High through Codex CLI 0.144.1. I compared no plugin, Ponytail, and RTK, with two runs per condition.
What the matched runs showed
| Condition | Mean verifier score | Mean tokens | Mean modeled cost | Mean rounds |
|---|---|---|---|---|
| No plugin | 78.85% | 6.660M | $5.281946 | 62.5 |
| Ponytail | 80.77% | -7.56% vs baseline | -8.87% | — |
| RTK | 76.92% | +13.20% vs baseline | +7.18% | +44.00% |
Ponytail used fewer tokens and cost less in this small sample, but took 13.51% longer. RTK used more tokens, cost more, and took 40.69% longer.
The important caveat is the sample size: n=2 per condition is not enough to estimate a causal plugin effect. In fact, the cost range divided by the mean was 43.25% without a plugin, 51.69% for Ponytail, and 30.78% for RTK. Those within-condition swings are much larger than the headline differences between conditions.
That means a single good run can easily produce a persuasive “saved X%” claim even when ordinary agent-trajectory variance is the simpler explanation.
The denominator is the real problem
The broader dataset contains 140 Codex CLI Medium/High runs, 10,365 agent rounds, 901,608,531 tokens, and $680.34 in modeled API cost. It contains no Tura runs.
The token and cost distributions were very different:
- Cached input: 96.46% of tokens, 63.91% of modeled cost
- New uncached input: 3.16% of tokens, 20.94% of cost
- Model output: 0.38% of tokens, 15.14% of cost
So a tool can compress one prompt fragment or one class of terminal output by a huge percentage while barely moving the full-task bill. It may also change the agent’s path: extra searches, retries, tool calls, or rounds can erase the local saving.
For example, the dataset contains 1,082 uniquely classified RTK-supported shell calls returning 1,458,927 tokens—only 0.1618% of all task tokens. Even a perfect 90% reduction of every eligible return corresponds to about 0.96% in directly attributable modeled cost savings.
A better evaluation checklist
Before accepting a token-saving claim, I now ask four questions:
- Was the same task repeated with and without the tool?
- Did an external verifier confirm the final result?
- Were retries, failures, orchestration, and tool calls included?
- Is the metric cost per verified successful task, rather than tokens removed from one step?
Compression can still be useful. The point is that local compression percentage is not a reliable substitute for end-to-end evidence.
The full analysis, assumptions, formulas, and reproducible data links are here:
https://turaai.net/blog#token-saving-plugins-are-mostly-stupid-idea
The open-source project is here:
https://github.com/Tura-AI/tura
Disclosure: I maintain Tura and wrote the linked analysis. This post was prepared with AI assistance and reviewed against the underlying benchmark data before publication.
Top comments (0)