Dan Luu's post on agentic coding, flagged in an 11-point Hacker News thread, reviews how test-driven agent loops change LLM performance on coding tasks.
What Agentic Coding Changes
Agentic setups run an LLM in a loop that writes code, runs tests, and iterates on failures. The process replaces single-shot generation with repeated verification steps.
This approach surfaces errors that static benchmarks miss. Luu notes that standard pass@k metrics do not capture the cost of multiple repair cycles.
Benchmark Numbers and Gaps
Current leaderboards report pass rates on HumanEval and similar suites. Agentic runs show lower effective success once test execution time and token spend are included.
Luu highlights that some models reach 80% pass rates in one shot but drop below 50% when forced to fix failing tests without human hints.
How to Run Agentic Tests Locally
Install a basic loop using the OpenAI or Anthropic SDK plus a test runner such as pytest. Feed the failing test output back into the model prompt until all tests pass or a step limit is reached.
Limit loops to 5-10 iterations to control cost. Track total tokens and wall time per task.
Tradeoffs Observed
- Higher final pass rates on multi-file tasks
- 3-8x increase in token usage versus single generation
- Greater variance across runs due to prompt sensitivity
Teams with strict latency budgets see limited gains. Projects with existing test suites benefit most.
Comparison With Non-Agentic Tools
| Approach | Pass Rate | Avg Tokens | Repair Loops |
|---|---|---|---|
| Single prompt | 65% | 800 | 0 |
| Agentic loop | 78% | 4200 | 4.2 |
| Cursor + tests | 72% | 2100 | 2.1 |
Cursor and Continue.dev sit between the two extremes by offering lighter repair suggestions.
Who Benefits Most
Developers maintaining large test suites gain the clearest signal. Researchers evaluating new models should add agentic metrics to avoid over-optimistic single-shot scores.
Teams without automated tests or with tight API budgets should stay with conventional prompting.
Practical Verdict
Agentic loops improve correctness on complex coding tasks but raise cost and latency. Add them only after baseline benchmarks are established.
Early data from the post suggests the biggest remaining gap is reliable test generation rather than code generation itself.
Top comments (0)