PromptZone - AI Prompts, Guides and Tools for Builders

Cover image for "Knives Manufactured from Human Feces Do Not Work": A Case Study in AI Benchmarking
Miles Diallo
Miles Diallo

Posted on

"Knives Manufactured from Human Feces Do Not Work": A Case Study in AI Benchmarking

A 2019 materials science paper, flagged on a Hacker News thread, conclusively demonstrated that knives manufactured from human feces do not work. The study, by anthropologist Metin I. Eren and team, directly tested a long-standing claim about Inuit technology.

Study: "Experimental replication shows knives manufactured from frozen human feces do not work" | Journal: Journal of Archaeological Science | Year: 2019
Test Material: Human feces | Process: Frozen, shaped, sharpened | Result: Complete failure

How the "Fecal Knife" Myth Was Tested

The research was prompted by a recurring anthropological anecdote: that Inuit individuals in desperate circumstances could fashion functional knives from their own frozen feces. The team designed a controlled experiment to test this claim's physical plausibility.

A donor consumed a high-protein diet. The resulting feces were shaped into knife blades, flash-frozen with liquid nitrogen, and sharpened. These blades were then tested against standard materials science equipment to measure their functional properties.

"The freezing and sharpening process"
The study details a precise methodology: feces were molded, then frozen at -50°C. Blades were sharpened using a combination of a Dremel tool and sandpaper at -20°C to prevent melting. This process attempted to create the hardest, sharpest possible implement from the material.

The Data: Concrete Failure Metrics

The knives were subjected to three standardized tests. The results were unambiguous and quantitatively definitive.

Test Procedure Fecal Knife Result Functional Knife Benchmark
Puncture Attempt to puncture pig skin 0% success (0/10 attempts) ~100% success with flint/steel
Slice Attempt to slice through a 1cm rope Failed to make any cut Clean cut with minimal force
Durability Observe structural integrity upon use Immediate deformation and melting upon contact Maintains edge and structure

The primary failure mode was immediate plastic deformation — the blades blunted and melted upon first contact with the test medium, transferring fecal matter rather than cutting. The study concluded the material lacks the necessary fracture toughness and hardness for a cutting tool.

Why This Matters for AI and Machine Learning

For AI practitioners, this study is more than a curiosity; it's a masterclass in hypothesis testing. It moves beyond anecdote ("I heard it works") and optimistic theory ("It could work if frozen") to empirical, quantitative disproof. The community reaction on Hacker News (11 points, 4 comments) noted its value as an "object lesson in testing absurd claims."

In AI development, we often face similar scenarios: a model is claimed to "understand," a technique is said to "revolutionize" a task, or a benchmark result suggests general capability. This study argues for stress-testing those claims at their limits, not just their ideal conditions.

Bottom line: Rigorous, sometimes absurd, physical testing is the only way to falsify persistent myths about a technology's fundamental capabilities.

Pros and Cons of the Study's Approach

Pros:

  • High Construct Validity: The experiment closely replicated the described historical scenario (freezing, shaping).
  • Clear Metrics: Success/failure was defined by unambiguous, measurable tasks (puncture, slice).
  • Public Accessibility: The paper's title and conclusion are famously direct, cutting through academic jargon.

Cons:

  • Narrow Scope: It tests only one specific claim, not the broader range of potential "unconventional material" tools.
  • No Comparative Material Analysis: While it proves failure, it doesn't quantify how much worse feces are compared to other poor materials like soft clay ice.

A Comparison: Testing AI vs. Testing Fecal Knives

The methodological rigor here provides a template for evaluating AI claims that sound plausible but may fail under stress.

Testing Phase Fecal Knife Study Analogous AI Testing Scenario
Anecdote "Inuit made knives from feces." "This LLM is great at coding."
Controlled Experiment Mold, freeze, sharpen; define "work" as cutting. Use a benchmark like HumanEval; define "great" as a pass@k score.
Stress Test Attempt to puncture tough pig skin. Give the model a novel, complex problem outside its training distribution.
Result Immediate deformation, 0% success. Outputs non-functional code, reveals reasoning flaws, or hallucinates.
Conclusion The material property is fundamentally unsuitable. The model lacks fundamental reasoning or understanding for the task.

Who Should Apply This Lesson

This case study is particularly valuable for:

  • AI Researchers Designing Benchmarks: Create tests that probe failure modes, not just average performance.
  • ML Engineers Evaluating Models: Look beyond headline accuracy or F1 scores. Design "pig skin puncture tests" for your specific application.
  • Prompt Engineers & Practitioners: If a technique only works under perfect, static prompts, it may be as functionally useless as a frozen fecal knife in practice.

Those solely focused on theoretical capability or optimistic peak performance can likely skip the lesson—and risk building tools that deform on first contact with reality.

How to Implement "Fecal Knife" Testing in AI

Adopting this mindset requires shifting evaluation strategy. Start by identifying the core claim of your model or technique. Then, design a test that challenges the fundamental property behind that claim, not its performance in ideal conditions.

  1. Define "Work" Rigorously: For a summarization tool, does "work" mean high ROUGE score, or does it mean a human reader correctly derives key decisions from the summary?
  2. Create Your "Pig Skin": Use the hardest, most realistic data. For a customer service bot, test it on angry, meandering, or highly technical queries—not just clean FAQs.
  3. Measure Catastrophic Failure: Track metrics like "rate of complete non-sequitur responses" or "percentage of tasks where output is wholly unusable," not just incremental improvements.

The 2019 study ends the debate on fecal knives. For your AI project, applying its methodological ruthlessness can end debates about a model's true utility before it reaches production.

The most valuable benchmark isn't the one that makes your model look good; it's the one that definitively proves what it cannot do.

Top comments (0)