A new paper titled "Breaking the 1.58-bit Barrier for Ternary LLMs" appeared on arXiv and quickly reached 182 points with 25 comments on Hacker News.
The work focuses on training large language models where every weight takes one of three values: -1, 0, or +1. This representation uses exactly log₂(3) ≈ 1.58 bits per parameter.
Model: Ternary LLM | Bits per weight: 1.58 | Parameters: variable | License: arXiv preprint
What It Is and How It Works
Ternary weights replace the usual 16-bit or 8-bit floating-point values with a simple three-state representation. During training, the optimizer learns a scaling factor per layer while keeping the weights strictly in {-1, 0, +1}.
The method introduces a straight-through estimator that allows gradients to flow through the non-differentiable quantization step. At inference time, matrix multiplications become additions and subtractions, eliminating most multiply-accumulate operations.
Measured Performance Numbers
The authors report that models trained with this scheme match the perplexity of full-precision baselines on standard language modeling benchmarks. Memory footprint drops by roughly 10× compared with BF16 weights.
Early HN commenters noted that the 1.58-bit figure beats previous 2-bit and 4-bit quantization results while preserving downstream task accuracy within 1 %.
| Quantization | Bits/weight | Relative memory | Accuracy drop |
|---|---|---|---|
| BF16 baseline | 16 | 1.0× | 0 % |
| 4-bit | 4 | 4.0× | 1–3 % |
| 2-bit | 2 | 8.0× | 3–8 % |
| Ternary (new) | 1.58 | 10.1× | <1 % |
How to Try the Approach
The paper supplies training code and configuration files on the arXiv page. Researchers can download the repository, install the provided PyTorch extensions, and reproduce the 1.58-bit training run on a single 8×A100 node.
No public inference engine yet supports the custom ternary kernels, but community ports for llama.cpp and vLLM are already under discussion in the HN thread.
Pros and Cons
- Memory reduction reaches 10× with negligible accuracy loss.
- Matrix multiplies simplify to additions, lowering power draw on edge devices.
- Current kernels lack optimized GPU support, so real-world speedups remain theoretical.
- Training requires the custom straight-through estimator; standard optimizers cannot be used directly.
Alternatives and Comparisons
Existing 4-bit quantization libraries such as GPTQ and AWQ already run on consumer GPUs. The new ternary method offers a further 2.5× memory cut but currently has no production inference stack.
BitNet b1.58 from Microsoft Research used a similar 1.58-bit scheme; the new paper improves training stability and reports higher zero-shot scores on the same model sizes.
Who Should Use This
Teams building on-device or cost-sensitive inference pipelines will benefit once kernels land. Researchers focused on extreme quantization should test the released training scripts immediately. Production teams needing plug-and-play speed should wait for optimized runtimes.
Bottom line: The paper demonstrates that 1.58-bit ternary weights can match full-precision accuracy, shifting the practical limit for local LLM deployment.
The next milestone will be production-grade kernels that turn the theoretical 10× memory saving into measurable tokens-per-second gains on consumer hardware.
Top comments (0)