# Can Gemma Know When It's Wrong?

> Published 2026-07-23 · https://www.promptzone.com/hyun_rao/can-gemma-know-when-its-wrong-3j7o

Cactus Hybrid surfaced on Hacker News with 46 points and 7 comments. The project fine-tunes Gemma to output both an answer and a calibrated that the answer is correct.

## What It Is / How It Works
The repo at [https://github.com/cactus-compute/cactus-hybrid](https://github.com/cactus-compute/cactus-hybrid) adds a lightweight uncertainty head on top of Gemma-2 weights. During training, the model learns to predict token-level and sequence-level probabilities of being factually wrong. At inference it returns the answer plus an explicit "uncertainty score" between 0 and 1.

The approach combines supervised fine-tuning on synthetic error examples with a small calibration dataset. No architectural changes beyond the added head are required.

## Benchmarks and Numbers
Early numbers shared in the thread show the hybrid model reduces over-confident wrong answers by 34 % on a 1 000-question subset of TriviaQA compared with base Gemma-2-9B. Average uncertainty score on correct answers sits at 0.18; on incorrect answers it rises to 0.71.

Latency overhead is 3–5 ms per token on an A100. Memory increase is under 200 MB.

## How to Try It
Clone the repository and run the provided inference script:
```bash
git clone https://github.com/cactus-compute/cactus-hybrid
pip install -r requirements.txt
python run_inference.py --model cactus-gemma-9b --prompt "What is the capital of Australia?"
```
The output includes both the answer and the uncertainty score. A Hugging Face space is also linked in the README for quick browser testing.

## Pros and Cons
- Pros: explicit uncertainty signal, minimal added compute, works with existing Gemma weights.
- Cons: still requires task-specific calibration data; performance on long-form generation not yet measured; 9 B model only so far.

## Alternatives and Comparisons
Standard temperature scaling and verbalized confidence prompting remain the most common baselines.

| Method              | Extra Training | Uncertainty Granularity | Overhead | Open Weights |
|---------------------|----------------|-------------------------|----------|--------------|
| Temperature scaling | No             | Sequence only           | 0 ms     | Yes          |
| Verbalized prompting| No             | Sequence only           | 0 ms     | Yes          |
| Cactus Hybrid       | Yes (LoRA)     | Token + sequence        | 3-5 ms   | Yes          |
| Conformal prediction| Yes            | Set-valued              | 10+ ms   | Varies       |

Cactus Hybrid is the only listed option that supplies per-token uncertainty without post-hoc sampling.

## Who Should Use This
Teams building retrieval-augmented systems or agent loops benefit most; the uncertainty score can trigger fallback retrieval or human review. Researchers studying calibration will find the released training code useful. Skip it if you only need one-off chat responses where raw speed matters more than error flagging.

## Bottom Line / Verdict
Cactus Hybrid demonstrates that a modest fine-tuning pass can give Gemma a usable internal "I don't know" signal without heavy architectural overhaul.

The project is still early, yet the released code and reported 34 % reduction in over-confident errors make it worth testing for any production pipeline that already runs Gemma locally.