Guide · Open source

How to use llama.cpp: install, run models and start a local API

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

llama.cpp is an open-source MIT-licensed project for running large language models and vision-language models on your own machine. It is written in C/C++, has no required dependencies, and supports a wide range of CPUs and GPUs. This guide shows you how to install it, download and run a model, and start the built-in API server so other tools can talk to it.

llama.cppggml-org/llama.cpp · ★ 133k · MIT Install options & downloads

Quick start

1Install llama.cpp. Run the curl installer on macOS or Linux, or the PowerShell installer on Windows. You can also build from source or use Docker.
2Check your hardware. Confirm your RAM or VRAM can hold the model you plan to use, since performance depends heavily on hardware and model size.
3Run a model with llama cli. Use llama cli with the -hf flag to download and run a GGUF model from Hugging Face in your terminal.
4Start the API server. Run llama serve with a model to launch the OpenAI-compatible API server and built-in web UI.
5Connect a client. Point an OpenAI-compatible client at the server address to send prompts from your own apps.

What llama.cpp is and who it suits

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup on a wide range of hardware, both locally and in the cloud. You run it yourself; it is not a managed cloud service.

It is a good fit for developers who want open-weight models without a cloud API, people with Apple silicon or other non-NVIDIA hardware who want GPU-accelerated inference, and teams that want a self-hosted OpenAI-compatible endpoint. It is less suited to users who want a polished consumer chat app with no setup, since the project points to command-line and build steps.

Runs on Apple silicon, x86 and RISC-V CPUs, and NVIDIA, AMD and other GPUs
Supports 1.5-bit through 8-bit integer quantization to reduce memory use
Includes a command-line tool, a built-in web UI and an OpenAI-compatible API server

Install llama.cpp on macOS, Linux or Windows

The project offers a few ways to get it onto your machine: an installer script, Docker, pre-built binaries from the releases page, or building from source. The README lists the installer commands below for macOS/Linux and Windows.

On macOS or Linux, run the curl installer. On Windows, run the PowerShell installer. If you prefer to build it yourself, clone the repository and follow the build guide in the docs folder.

Installer: visit https://llama.app and follow the instructions
Docker: see the Docker documentation in the repository
Pre-built binaries: download from the releases page on GitHub
Build from source: clone the repository and read docs/build.md

# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh

# Windows (PowerShell)
irm https://llama.app/install.ps1 | iex

# Build from source
git clone https://github.com/ggml-org/llama.cpp

Run your first model with llama cli

Once installed, you can download and run a model directly from Hugging Face with the llama cli tool. The example in the README uses a small GGUF model from the ggml-org account on Hugging Face. The command opens an interactive session in your terminal.

The README shows a vision-language model session running in llama cli, so the same tool can handle image input for models that support it.

Pick a model that fits your RAM or VRAM; performance depends heavily on hardware and model size
Start with a small model to confirm the setup works

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

Start the OpenAI-compatible API server

To use llama.cpp from other apps, run llama serve. It launches an OpenAI-compatible API server and a built-in web UI that you can open in a browser. This is the option teams use when they want a self-hosted endpoint.

Point any client that supports the OpenAI API format at the server address that llama serve reports when it starts.

The server includes a built-in web UI for chatting with the model
The API follows the OpenAI-compatible format
Documentation for the server is in tools/server/README.md

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Choose the right hardware backend

llama.cpp uses different backends depending on your hardware. Apple silicon uses Metal, NVIDIA GPUs use CUDA, AMD GPUs use HIP, and Vulkan and SYCL are also supported. The project's backend table lists which device each one targets.

If a model is larger than your GPU memory, the project supports CPU+GPU hybrid inference, which partially accelerates the model. Performance still depends on your hardware and model size, so test with a model you can comfortably load.

Apple silicon: Metal and Accelerate
NVIDIA GPU: CUDA
AMD GPU: HIP
Intel GPU: SYCL; Vulkan works on GPUs generally
Multi-GPU and performance tips are in docs/multi-gpu.md and docs/development/token_generation_performance_tips.md

Troubleshoot and learn more

If a model runs slowly or fails to load, check the performance troubleshooting notes in the docs folder and confirm that the model fits in your memory. The project's documentation index covers tools, building, Docker, Android, and model formats.

Read CONTRIBUTING.md in the repository if you want to help with the project.

Performance troubleshooting: docs/development/token_generation_performance_tips.md
Models: docs/models.md
Build instructions: docs/build.md

What people use it for

Running open-weight models locally without a cloud API
Hosting a self-hosted OpenAI-compatible endpoint for a team
Running vision-language models with llama cli
Experimenting with quantized models on limited memory
Using GPU acceleration on Apple silicon or other non-NVIDIA hardware

Frequently asked questions

Is llama.cpp free?

llama.cpp is open source and released under the MIT license, as stated on the project page. You can use, modify and distribute it under that license.

What do I need to install llama.cpp?

The source lists no prerequisites for installation. You can use the installer scripts, Docker, pre-built binaries or build from source by cloning the repository.

How do I run llama.cpp on Apple silicon?

Apple silicon is a first-class target in llama.cpp, optimized with ARM NEON, Accelerate and Metal frameworks. Install it with the macOS installer or build from source, then run llama cli or llama serve with a model.

Does llama.cpp have a web interface?

Yes. llama serve includes a built-in web UI that you can open in a browser while the API server is running.

How does llama.cpp compare to Ollama or LM Studio?

Ollama, LM Studio, vLLM and MLC LLM are comparable projects. llama.cpp is a command-line and build-oriented project with an API server, so it suits users comfortable with terminal setup more than those who want a polished chat app.

Why is my model slow or failing to load?

Performance depends heavily on hardware and model size. Check that the model fits in your RAM or VRAM, try a smaller or more quantized model, and read the performance troubleshooting notes in the docs folder.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

llama.cpp suits developers, researchers and teams who are comfortable with the terminal and want to run open-weight models on their own hardware, including Apple silicon and non-NVIDIA GPUs. People who want a polished consumer chat app with no setup, or a managed cloud service, should pick a tool such as LM Studio, Ollama or a hosted API instead.

Install options & downloads