Skip to content
Get packs

llama-cpp-python

Python bindings that let you run llama.cpp language models from Python code or a local OpenAI-style server.

GitHub Download v0.3.36-hip-radeon
Stars
10.7k
Forks
1.5k
Licence
MIT
Language
Python
Last update
1 months ago
Type
Library

What it is

This package wraps llama.cpp, a program for running large language models on your own computer, so Python code can load a model and generate text with it. It offers a high-level Python API, compatibility with LangChain and LlamaIndex, and an OpenAI-compatible web server you can point other tools at. People use it to run models locally instead of calling a cloud service.

  • Load a local GGUF model in Python and generate text with the high-level API
  • Run an OpenAI-compatible web server on your machine for local apps to call
  • Use it as the local model inside a LangChain or LlamaIndex pipeline
  • Set up a local code-completion backend that replaces a cloud Copilot-style service, as the README describes

Great for

  • Python developers who want to run a language model locally from their own code
  • Developers who want a local server that mimics the OpenAI API for existing tools
  • Users building LangChain or LlamaIndex apps who want a local model backend
  • People who want to run models on specific hardware such as NVIDIA, AMD, Apple Metal, or Vulkan GPUs using the documented build options

Not ideal for

  • People who only want a ready-made chat app; this is a library and server for developers, not a graphical application, based on the README
  • Users with no Python or compiler setup, since the default install builds llama.cpp from source and needs a C compiler
  • Anyone needing a guarantee of performance on their hardware; the README does not give benchmarks
Step-by-step guide: how to use llama-cpp-pythonInstall, first run, examples, tips and FAQs
llama.cpp vs llama-cpp-pythonllama-cpp-python vs Ollama

How to get it

  1. 1

    Install with a package manager

    You'll need: Python 3.8+ · C compiler (gcc or clang on Linux, Visual Studio or MinGW on Windows, Xcode on macOS)

    Install from PyPI (builds llama.cpp from source)

    pip install llama-cpp-python

    Install a pre-built CPU wheel

    pip install llama-cpp-python \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

    Install with CUDA support

    CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
  2. 2

    Run it

    • Install the package with pip using one of the commands above
    • Obtain a GGUF model file (the README does not specify a source)
    • Load the model in Python with the high-level API or start the OpenAI-compatible server
    • Send a test completion request from Python or a client tool
  3. 3

    Get the most out of it

    • Use CMAKE_ARGS to enable a hardware backend such as CUDA, Metal, HIP, Vulkan, or BLAS that matches your machine
    • Use the pre-built wheel index if you want to skip compiling llama.cpp locally
    • Configure multiple models through the server's configuration and multi-model support described in the docs
    • Use the function calling and vision API support in the server if your model supports them
    • Read the macOS Metal install guide if you are on Apple Silicon

Downloads

Source code (.zip)main branch · straight from GitHubRelease v0.3.36-hip-radeon2 files incl. llama_cpp_python-0.3.36-py3-none-win_amd64.whlOfficial website & docsllama-cpp-python.readthedocs.io

Strengths

  • Gives an OpenAI-style interface, so many existing tools can connect with little change
  • Supports several hardware backends, including CUDA, Metal, HIP, Vulkan, and SYCL, through build options
  • Released under the MIT license with active documentation and a test badge in the README

Trade-offs

  • Installing from source requires a C compiler and CMake-based build, which can fail and needs troubleshooting
  • Pre-built wheels are listed as options for specific platforms and CUDA or ROCm versions, so availability depends on your setup
  • Performance and model support depend on the underlying llama.cpp project, which the README does not detail

How it compares

ProjectStarsTypeLevelUpdatedBest for
llama-cpp-python (this)10.7kLibraryintermediate1 months agoPython developers who want to run a language model locally from their own code
llama.cpp133kLibraryintermediate4 days agoDevelopers who want to run open-weight models locally without a cloud API
ollama186kAppbeginner4 days agoDevelopers who want a local model API for their own apps

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.