Petals
Run big language models by sharing the work across volunteers' GPUs, BitTorrent-style.
- Stars
- 10.7k
- Forks
- 650
- Licence
- MIT
- Language
- Python
- Last update
- 2.2 years ago
- Type
- Library
No commits in over a year — check for forks or alternatives before adopting.
What it is
Petals lets you load a small part of a very large language model on your own computer and connect to a network of other people who host the remaining parts. You then run text generation or fine-tuning as if the whole model were local, using PyTorch and Hugging Face Transformers. It is aimed at people who want to use models too big for their own hardware.
- Generate text with a large model such as Llama 3.1 405B using a few lines of Python
- Prompt-tune a large model for a classification task using the linked Colab tutorials
- Host a part of a model on your GPU to help serve the public swarm
- Run a private swarm among trusted machines for sensitive data
- Inspect hidden states or execute custom paths through a model
Great for
- Developers who want to experiment with large open models like Llama 3.1 or BLOOM without owning the hardware to run them alone
- Researchers prompt-tuning large models on their own tasks
- People with a spare GPU who want to contribute capacity to a community network
- Users comfortable with Python and Transformers who want to inspect hidden states or run custom paths through a model
Not ideal for
- People who need strict privacy for sensitive data, since prompts are processed by other people's machines in the public swarm; the README suggests a private swarm instead
- Users who need fast, reliable responses for production workloads, since the README reports single-batch speeds of up to 6 tokens/sec for Llama 2 (70B)
- Non-programmers looking for a ready-made chat app, though the project links a separate chatbot website
- Anyone needing a model they lack access to: Llama models require requesting weights from Meta first
How to get it
- 1
Install with a package manager
You'll need: Python · PyTorch · Hugging Face account and huggingface-cli login for gated models like Llama · Conda (for the Linux server path shown in the README)
Install the Petals package from GitHub
pip install git+https://github.com/bigscience-workshop/petalsLog in to Hugging Face (needed for Llama)
huggingface-cli login - 2
Run it
- Install Petals with pip as shown in the README
- Log in with huggingface-cli if using Llama and you have been granted access
- Load a model with AutoDistributedModelForCausalLM.from_pretrained using a model name from health.petals.dev
- Call model.generate on tokenized input to get output text
- 3
Get the most out of it
- Try the Colab getting-started notebook before installing locally
- Use the prompt-tuning example notebooks for classification or chatbot personas
- Check model availability and network health at health.petals.dev
- Use --public_name to be credited on the swarm monitor if you host 10+ blocks
- Read the wiki guides for private swarms and running custom models
Downloads
Strengths
- Lets you use models far larger than your local hardware can hold, by splitting layers across the network
- Works with familiar tools: PyTorch and Hugging Face Transformers APIs
- Supports fine-tuning and sampling methods, not only inference
Trade-offs
- Depends on other volunteers' machines, so performance and availability vary with the network
- Privacy depends on the swarm; sensitive data needs a private swarm the README says you must set up yourself
- The README's install instructions are partly tied to older CUDA and PyTorch versions, and the last push shown is from 2024-09-07, so it may need adjustment
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| petals (this) | 10.7k | Library | intermediate | 2.2 years ago | Developers who want to experiment with large open models like Llama 3.1 or BLOOM without owning the hardware to run them alone |
| llama.cpp | 133k | Library | intermediate | 4 days ago | Developers who want to run open-weight models locally without a cloud API |
| ollama | 186k | App | beginner | 4 days ago | Developers who want a local model API for their own apps |
| onnxruntime | 22.2k | Library | intermediate | 4 days ago | Developers who want to deploy a trained model inside an application or service |
| annotated_deep_learning_paper_implementations | 67.6k | Learning resource | intermediate | 4 days ago | Students learning how transformers, GANs, or optimizers work internally |
| nni | 14.4k | Library | intermediate | 2.3 years ago | Researchers comparing hyperparameter tuning algorithms such as TPE, Hyperband, or Bayesian methods |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.