Ollama vs Petals

Ollama

• Developers who want a local model API for their own apps
• People who want to chat with open models without sending prompts to a hosted service
• Users who want to connect a coding assistant to a local or self-chosen model
• Students experimenting with different open models from a terminal

Petals

• Developers who want to experiment with large open models like Llama 3.1 or BLOOM without owning the hardware to run them alone
• Researchers prompt-tuning large models on their own tasks
• People with a spare GPU who want to contribute capacity to a community network
• Users comfortable with Python and Transformers who want to inspect hidden states or run custom paths through a model

Ollama

• Simple install and run commands across macOS, Windows, Linux and Docker
• Official Python and JavaScript libraries plus a documented REST API
• Large community with many integrations listed in the README

Petals

• Lets you use models far larger than your local hardware can hold, by splitting layers across the network
• Works with familiar tools: PyTorch and Hugging Face Transformers APIs
• Supports fine-tuning and sampling methods, not only inference

Ollama

• People who need the most capable closed frontier models, since this focuses on open models
• Users with limited disk space or memory, because model files can be large and need capable hardware (the README does not give specific requirements)
• Non-technical users who want a polished desktop app with no terminal use, although the README mentions a download option for macOS and Windows

Petals

• People who need strict privacy for sensitive data, since prompts are processed by other people's machines in the public swarm; the README suggests a private swarm instead
• Users who need fast, reliable responses for production workloads, since the README reports single-batch speeds of up to 6 tokens/sec for Llama 2 (70B)
• Non-programmers looking for a ready-made chat app, though the project links a separate chatbot website
• Anyone needing a model they lack access to: Llama models require requesting weights from Meta first

Ollama

• The README is truncated and does not state hardware requirements or model sizes
• Some features, such as cloud access and specific integrations, depend on Ollama's own services and documentation that is only partly described here
• Model quality and licensing depend on each individual model, not on Ollama itself

Petals

• Depends on other volunteers' machines, so performance and availability vary with the network
• Privacy depends on the swarm; sensitive data needs a private swarm the README says you must set up yourself
• The README's install instructions are partly tied to older CUDA and PyTorch versions, and the last push shown is from 2024-09-07, so it may need adjustment