Cactus Hybrid surfaced on Hacker News with 46 points and 7 comments. The project fine-tunes Gemma to output both an answer and a calibrated that the answer is correct.
What It Is / How It Works
The repo at https://github.com/cactus-compute/cactus-hybrid adds a lightweight uncertainty head on top of Gemma-2 weights. During training, the model learns to predict token-level and sequence-level probabilities of being factually wrong. At inference it returns the answer plus an explicit "uncertainty score" between 0 and 1.
The approach combines supervised fine-tuning on synthetic error examples with a small calibration dataset. No architectural changes beyond the added head are required.
Benchmarks and Numbers
Early numbers shared in the thread show the hybrid model reduces over-confident wrong answers by 34 % on a 1 000-question subset of TriviaQA compared with base Gemma-2-9B. Average uncertainty score on correct answers sits at 0.18; on incorrect answers it rises to 0.71.
Latency overhead is 3–5 ms per token on an A100. Memory increase is under 200 MB.
How to Try It
Clone the repository and run the provided inference script:
git clone https://github.com/cactus-compute/cactus-hybrid
pip install -r requirements.txt
python run_inference.py --model cactus-gemma-9b --prompt "What is the capital of Australia?"
The output includes both the answer and the uncertainty score. A Hugging Face space is also linked in the README for quick browser testing.
Pros and Cons
- Pros: explicit uncertainty signal, minimal added compute, works with existing Gemma weights.
- Cons: still requires task-specific calibration data; performance on long-form generation not yet measured; 9 B model only so far.
Alternatives and Comparisons
Standard temperature scaling and verbalized confidence prompting remain the most common baselines.
| Method | Extra Training | Uncertainty Granularity | Overhead | Open Weights |
|---|---|---|---|---|
| Temperature scaling | No | Sequence only | 0 ms | Yes |
| Verbalized prompting | No | Sequence only | 0 ms | Yes |
| Cactus Hybrid | Yes (LoRA) | Token + sequence | 3-5 ms | Yes |
| Conformal prediction | Yes | Set-valued | 10+ ms | Varies |
Cactus Hybrid is the only listed option that supplies per-token uncertainty without post-hoc sampling.
Who Should Use This
Teams building retrieval-augmented systems or agent loops benefit most; the uncertainty score can trigger fallback retrieval or human review. Researchers studying calibration will find the released training code useful. Skip it if you only need one-off chat responses where raw speed matters more than error flagging.
Bottom Line / Verdict
Cactus Hybrid demonstrates that a modest fine-tuning pass can give Gemma a usable internal "I don't know" signal without heavy architectural overhaul.
The project is still early, yet the released code and reported 34 % reduction in over-confident errors make it worth testing for any production pipeline that already runs Gemma locally.
Top comments (0)