Run AI models on your laptop: Ollama, llama.cpp, GPT4All and friends compared

You don't need a cloud account to use a large language model. Open models can run on an ordinary laptop, and when they do your prompts and documents never leave the machine. The trade-off is that you supply the hardware, and smaller local models are usually less capable than the biggest hosted ones. This guide compares the open-source projects in our catalogue that make local AI practical, from one-command runtimes to full chat apps.

Two layers: the engine and the interface

Local AI tools tend to do one of two jobs. A runtime (or engine) loads model files and does the actual computation, usually exposing a local API that other apps can call. An interface gives you a chat window, document upload and history. Some projects bundle both. Knowing which layer a project covers saves a lot of confusion.

The runtimes

Ollama is the usual first stop. It installs as a normal app, downloads models with one command and runs them from your terminal. It also exposes a local API, which is why so many other tools, from chat front ends to coding agents, can plug into it. Our analysis rates it beginner-friendly; the main limits are your hardware and the size of the model downloads.

llama.cpp is the engine underneath much of the local AI world. It's written in plain C/C++ with no external dependencies and supports a long list of hardware back ends, including Apple silicon and NVIDIA GPUs. It can split work between CPU and GPU and run quantised models, which are compressed versions that need much less memory. It's mainly a command-line and programming tool, so it suits people who are happy to tune settings.

LocalAI aims to be a drop-in replacement for cloud APIs on your own hardware. If you already have code that talks to an OpenAI-style API, pointing it at LocalAI is the appeal. It is modular, pulling in only the back ends a model needs, and it expects some comfort with Docker or the command line.

OllamaOpen sourceThe easiest on-ramp: install, pull a model, run.★ 186k · MITllama.cppOpen sourceThe high-performance engine many other tools build on.★ 133k · MITLocalAIOpen sourceAn OpenAI-compatible API you host yourself.★ 50.1k · MIT

The chat apps

If you'd rather not touch a terminal, start with a desktop app. GPT4All runs models locally without an internet connection, API keys or a dedicated GPU, and its LocalDocs feature lets you ask questions about files on your computer. AnythingLLM is broader: it can connect to local or cloud models, manage document collections and build simple agents, with multi-user features in its Docker version.

For a shared, browser-based setup, LibreChat gives you a ChatGPT-style site with user accounts, agents and support for many providers, local models included. It's the most involved to set up, since it runs as several containers. NextChat is lighter: a cross-platform chat interface that stores data in your browser, though you bring your own API keys or a compatible local endpoint.

GPT4AllOpen sourceA private desktop chat app that runs fully offline.★ 77.3k · MITAnythingLLMOpen sourceChat with your documents using local or cloud models.★ 67.8k · MITLibreChatOpen sourceA self-hosted, multi-user chat site.★ 46.6k · MITOpen NotebookOpen sourceA self-hosted research notebook that works with local models.★ 41.3k · MIT

Our AI directory also lists LM Studio and Jan, two desktop apps for local models, and Open WebUI, a popular self-hosted chat front end. They don't appear in our open-source section, which only covers MIT-licensed repositories, but they're worth knowing about. LM Studio offers a friendly interface for downloading and comparing models; Jan is fully open source and can mix local and cloud models.

LM StudioAI toolA friendly desktop app for finding and chatting with local models.FreeJanAI toolAn open-source desktop assistant that works offline.Open sourceOpen WebUIAI toolA feature-rich self-hosted chat interface; check its licence terms.Open source

Will it run on my machine?

Memory is the main constraint. A model has to fit in your RAM (or your GPU's memory) to run at a usable speed, and larger models need more. Quantised versions of a model trade a little quality for a big reduction in memory, which is why they're the default in most local tools. Laptops with Apple silicon do well because the CPU and GPU share memory.

• Start small. Try a compact model first, check the speed, then move up.
• Close heavy apps. Browsers with many tabs compete for the same memory.
• Check before you download. llmfit inspects your CPU, RAM and GPU and suggests which models are likely to run well.
• On a Mac, MLX is Apple's array framework for machine learning on Apple silicon, aimed at people building or fine-tuning models rather than just chatting.

llmfitOpen sourceTells you which models your hardware can handle.★ 38.6k · MITMLXOpen sourceMachine learning on Apple silicon, for builders.★ 29k · MIT

Which should you pick?

• You want to try local AI tonight with minimal fuss: Ollama, or GPT4All if you prefer a window to a terminal.
• You want to chat with your own files privately: GPT4All's LocalDocs or AnythingLLM.
• You're building an app and want a local API: Ollama for simplicity, LocalAI or llama.cpp's server for OpenAI-style compatibility.
• You want a shared chat site for a household or small team: LibreChat on top of Ollama.
• You want to squeeze every bit of performance out of unusual hardware: llama.cpp.