GateGPT implements a Transformer with KV cache that reaches 56k tokens per second on an FPGA running at 80 MHz, according to discussion flagged on Hacker News.
The result was shared via a recent Hacker News thread that drew 27 points and 8 comments.
Model: GateGPT | Speed: 56k tokens/s | Hardware: FPGA | Clock: 80 MHz
What GateGPT Delivers on FPGA Hardware
GateGPT maps the attention KV cache directly onto FPGA fabric. The design keeps matrix operations and memory access on-chip at a fixed 80 MHz clock.
This removes the need for high-frequency GPU schedulers while sustaining high token throughput for inference workloads that reuse cached keys and values.
Key Performance Metrics
The reported figure of 56k tokens per second comes from a single FPGA implementation. No batch-size or sequence-length details were provided in the source post.
Early HN comments noted the low clock rate relative to typical GPU boost clocks above 1 GHz.
Tradeoffs of FPGA Inference
- Fixed 80 MHz clock limits peak frequency but simplifies power and cooling requirements.
- KV-cache mapping reduces external memory traffic compared with standard GPU attention kernels.
- Reconfiguring the FPGA for new model sizes requires synthesis time that GPUs avoid.
Comparing FPGA to GPU Alternatives
Standard GPU inference engines such as vLLM or TensorRT-LLM target consumer and data-center cards. The table below places the reported GateGPT numbers against typical published throughputs for similar KV-cache workloads.
| Hardware | Tokens/s (KV cache) | Clock | Notes |
|---|---|---|---|
| GateGPT FPGA | 56,000 | 80 MHz | Single reported result |
| A100 (FP16) | 8,000–15,000 | ~1.4 GHz | vLLM typical figures |
| RTX 4090 | 4,000–9,000 | ~2.5 GHz | TensorRT-LLM batch=1 |
GateGPT shows higher tokens per clock cycle than the GPU baselines listed.
Who Should Use This Approach
Teams building fixed-function inference pipelines with stable model architectures can benefit. Researchers exploring low-power or deterministic latency setups may also test the design.
Users who need frequent model updates or large-batch training should continue with GPU frameworks.
Final Assessment
GateGPT demonstrates that an 80 MHz FPGA can exceed common GPU token rates for KV-cache inference when the architecture stays simple and on-chip.
Bottom line: The result highlights a viable path for high-throughput inference on modest FPGA hardware when model changes remain infrequent.
Top comments (0)