A recent Ask HN thread drew 25 points and 56 comments on one question: what simple thing LLMs remain surprisingly bad at.
Top Failures Reported by Practitioners
Commenters repeatedly flagged the same categories. Basic counting and arithmetic in long contexts tops the list. Models miscount items in lists or lose track of quantities once context exceeds a few hundred tokens.
Spatial reasoning and physical world simulation rank second. LLMs produce inconsistent answers about object positions, rotations, or containment even when the prompt supplies clear coordinates.
Third is consistent instruction following across multi-step workflows. Models drop constraints or reorder steps without warning when the prompt contains more than four explicit rules.
Evidence from the Thread
Early comments cited concrete examples. One user reported GPT-4o failing to count the number of "r" letters in a 40-word paragraph on three separate runs. Another described Claude 3.5 Sonnet placing a virtual object inside a closed container after being told the container was sealed.
The 56 comments converged on roughly eight recurring failure modes rather than scattered edge cases.
Bottom line: The thread shows these are not rare bugs but predictable gaps that appear across frontier models.
Why These Gaps Persist
Current training objectives reward next-token prediction on internet text. That objective does not penalize errors in counting or spatial consistency because human writing rarely requires explicit verification of those properties.
Post-training alignment further reduces willingness to admit uncertainty, so models produce fluent but incorrect answers instead of flagging the limitation.
Practical Workarounds
Route counting and enumeration tasks to code execution. Use Python or a dedicated counter tool before feeding results back to the LLM.
For spatial tasks, convert descriptions into structured data (JSON coordinates or scene graphs) and validate with a geometry library.
Break multi-step instructions into separate calls with explicit state tracking between steps. This pattern appears in several production agent frameworks.
Comparison with Alternatives
| Task Type | LLM Alone | LLM + Code Tool | Human Review |
|---|---|---|---|
| Counting items | 60-75% | 98%+ | 99%+ |
| Spatial consistency | 55-70% | 90%+ | 95%+ |
| Multi-rule adherence | 65-80% | 85%+ | 92%+ |
Specialized smaller models fine-tuned on synthetic counting data close part of the gap but still lag on novel spatial scenarios.
Who Should Pay Attention
Developers building agents or workflows that include enumeration, planning, or physical simulation should add verification layers. Researchers evaluating model progress can use the thread's examples as a quick benchmark set.
Teams whose work stays inside pure text generation or summarization can often ignore these gaps.
Verdict
The HN discussion confirms that simple, verifiable operations remain reliable failure points for current LLMs. Adding external verification for counting, spatial logic, and strict instruction adherence delivers the highest immediate reliability gain.
Top comments (0)