Petals surfaced on Hacker News with 92 points and 29 comments for its approach to running large language models across volunteer GPUs in a decentralized swarm.
The system splits model layers across participants so no single machine needs the full parameter set in memory.
What It Is / How It Works
Petals treats inference as a peer-to-peer task. Each node hosts one or more transformer blocks and forwards activations to the next peer in the chain.
Users join the swarm by running a lightweight client that discovers other nodes via a DHT, similar to BitTorrent trackers.
The project site at https://petals.dev/ provides the client and lists active public swarms for models such as Llama-2-70B and Falcon-180B.
Benchmarks and Resource Numbers
Early swarm tests show 70B-class models generating tokens at 1–4 tokens per second when 8–12 consumer GPUs participate.
A single RTX 3090 node uses roughly 8–10 GB VRAM to host two layers of a 70B model.
Latency rises with network distance; participants on the same continent report median round-trip times under 80 ms.
How to Try It
Install the client with pip install petals then run python -m petals.cli.run_server --model bigscience/bloom-petals.
The command automatically registers the node and begins serving layers.
For inference, connect via the Petals Python client or the provided Gradio demo at the project site.
Pros and Cons
- Pros: Runs models larger than local VRAM allows; no central server costs; works on consumer hardware.
- Cons: Speed depends on peer availability and internet latency; public swarms can be unstable; no built-in fine-tuning support.
Alternatives and Comparisons
| Feature | Petals | Hugging Face TGI | vLLM (single node) |
|---|---|---|---|
| Max model size | 100B+ (swarm) | Limited by GPU memory | Limited by GPU memory |
| Hardware requirement | Multiple 8–12 GB GPUs | One 24–80 GB GPU | One 24–80 GB GPU |
| Decentralized | Yes | No | No |
| Tokens per second | 1–4 (70B swarm) | 20–60 (A100) | 30–80 (A100) |
Other options include Together.ai for paid API access and llama.cpp for fully local quantized runs.
Who Should Use This
Researchers testing 70B+ models without access to multi-GPU servers benefit most. Developers needing consistent low-latency inference should skip it and use single-node tools instead.
Bottom line: Petals fills the gap between fully local quantized models and expensive cloud clusters for occasional large-model experiments.
The project shows decentralized inference can move from theory to usable speeds when enough participants contribute GPUs.
Top comments (0)