Skip to content
Get packs

llama.cpp Local Inference

Guides the agent to set up and run local LLMs on CPUs, Macs and non-NVIDIA GPUs using llama.cpp.

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

When installed, the agent knows how to run open-source language models on ordinary hardware such as a laptop CPU, an Apple Silicon Mac, or AMD/Intel GPUs, without needing an NVIDIA card. It explains how to download or convert a model into the compact GGUF format, pick a quantization level that trades quality for memory and speed, and start the llama.cpp command-line tool or an OpenAI-compatible local server. It also recommends alternatives such as TensorRT-LLM or vLLM when NVIDIA GPUs are available.

Great for

  • Developers who want a private, local OpenAI-style API for testing models
  • Mac users running open models on Apple Silicon
  • People deploying small models on edge devices or machines without CUDA

Skip it if

  • Production datacenter serving at very high throughput, where the skill itself recommends TensorRT-LLM or vLLM instead
  • Users who expect guaranteed-accurate benchmark figures: the performance tables are unsourced and hardware-specific

Try asking

  • “Set up llama.cpp on my M2 Mac and run a 7B chat model locally”
  • “Start an OpenAI-compatible llama-server on port 8080 with a Q4_K_M model”
  • “Which GGUF quantization should I use to fit a 13B model in 16 GB of RAM?”
  • “Help me constrain model output to valid JSON using a grammar file”
Step-by-step guide: how to use llama.cpp Local InferenceInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    llama-cpp.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip llama-cpp.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip llama-cpp.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Verify commands and flags against the current llama.cpp release, since the skill's examples may drift from upstream.
  • Download models only from sources you trust, and check model licences before use.
  • If you bind the server to 0.0.0.0 as the example does, restrict access with a firewall, because the server has no authentication by default.

What's inside

  • references/optimization.md
  • references/quantization.md
  • references/server.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.