Skip to content
Get packs

Guide · Agent skill

How to use the Hugging Face Local Models agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The Hugging Face Local Models skill teaches an AI agent to find open models packaged as GGUF files on the Hugging Face Hub, pick a suitable compression level for your hardware, and launch them with llama.cpp. It is aimed at developers and technical users who want to run models on their own machine or serve them to an app through a local OpenAI-compatible endpoint. By the end of this guide you will know what the skill does, how to install it, how to phrase requests so it triggers, and what you need on your system first.

Hugging Face Local Modelshuggingface/skills · Apache-2.0

Quick start

  1. 1Download and install the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/, or into a project's .claude/skills/ folder, or upload it under Settings, Capabilities, Skills in the Claude apps.
  2. 2Install llama.cpp. Install it with brew install llama.cpp on macOS or winget install llama.cpp on Windows, or clone and build it from source with make.
  3. 3Log in for gated repos. Run hf auth login if you need gated or private models. The skill does not set up authentication for you.
  4. 4Describe your goal and hardware. Ask the agent for a model for a specific task and state your RAM, VRAM or GPU type so it can choose a size and quant.
  5. 5Launch the model. Accept the llama-cli or llama-server command the agent gives you, using -hf with the repo and quant, or --hf-repo with --hf-file for custom filenames.
  6. 6Test the endpoint. Send a chat request to http://localhost:8080/v1/chat/completions with curl, or point your app at that local endpoint.

What is an agent skill, and what does this one do?

An agent skill is a folder containing a SKILL.md file and optional reference files that an AI agent can load when a task matches the skill's description. This one is named huggingface-local-models and its description says it is for selecting models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm.

Once active, the agent follows a fixed workflow: it searches the Hub for llama.cpp-compatible GGUF repos, opens the repo's local-app page to find the recommended quant, confirms exact .gguf filenames, and then gives you a launch command using llama-cli or llama-server. If a repo has no GGUF files, the skill can convert Transformers weights to GGUF.

Install the skill

Option 1: download the ZIP for this skill from this page and unzip it into ~/.claude/skills/ for all your projects, or into a project's .claude/skills/ folder for one project only.

Option 2: in the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.

Whichever route you use, the skill folder should contain SKILL.md and the references folder with hardware.md, hub-discovery.md and quantization.md.

Install llama.cpp and the Hugging Face CLI

The skill assumes llama.cpp is already installed. The source lists Homebrew and winget as install routes, and building from source is the fallback. Gated or private repositories also need the Hugging Face CLI and an account token, which you set up by logging in.

  • macOS with Homebrew: brew install llama.cpp
  • Windows with winget: winget install llama.cpp
  • Any platform: clone https://github.com/ggml-org/llama.cpp, then run make inside the folder
  • Gated repos: hf auth login
brew install llama.cpp winget install llama.cpp git clone https://github.com/ggml-org/llama.cpp cd llama.cpp make hf auth login

How to trigger the skill with natural requests

You do not need special syntax. Describe the outcome you want in plain language that mentions local running, GGUF, llama.cpp, or your hardware. The skill's description covers finding GGUFs, quant selection, running servers, exact file lookup, conversion, and OpenAI-compatible serving, so requests on those topics are the ones it is built for.

Include your memory or GPU details when you can. The skill's quant guidance depends on how much RAM or VRAM you have.

  • Find a GGUF model under 24B parameters that runs well on a 16 GB Mac.
  • Which quant of Qwen should I use on an 8 GB NVIDIA card?
  • Start a llama-server for a coding model and give me the curl command to test it.
  • I have a Transformers model with no GGUF. Convert it and quantize it to Q4_K_M.
  • Set up a local OpenAI-compatible endpoint I can point my app at.

Run a model and test the local server

The skill can launch a model straight from the Hub with the -hf flag, using the repo name and the quant after a colon. If the repo uses unusual filenames, it falls back to --hf-repo together with --hf-file. The server listens on localhost port 8080 by default in the smoke test, and it accepts OpenAI-style chat requests at /v1/chat/completions.

The example repo and quant below come from the skill's own documentation. Replace them with whatever the agent finds for your needs.

  • Run as a chat tool with llama-cli -hf <repo>:<QUANT>
  • Run as a server with llama-server -hf <repo>:<QUANT>
  • Pin an exact file with --hf-repo and --hf-file, and set context size with -c
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{"messages": [{"role": "user", "content": "Write a limerick about exception handling"}]}'

How the agent picks a quant

A quant is a compression level for a model. Smaller quants need less memory and run faster but lose some accuracy. The skill's rules are to use the quant that the Hub's local-app page marks as compatible, keep repo-native labels such as UD-Q4_K_M, and default to Q4_K_M unless the page or your hardware suggests otherwise.

For code or technical work, the skill prefers Q5_K_M or Q6_K when memory allows. For tighter RAM or VRAM budgets it considers Q3_K_M, Q4_K_S, or repo-specific IQ and UD variants. Files named mmproj-*.gguf are projector weights, not the main model, and should not be treated as the checkpoint.

Hardware notes for acceleration

The hardware reference covers building llama.cpp with acceleration for your platform. On Apple Silicon the build uses GGML_METAL=1 and -ngl 99 offloads layers to the GPU. On NVIDIA the build uses GGML_CUDA=1, and the layer count is adjusted to fit your card. AMD uses LLAMA_HIP=1 for ROCm. For CPU-only machines, the skill suggests matching the thread count to physical cores, not logical threads.

If a model is too large for your GPU, the reference describes splitting layers between GPU and CPU, and multi-GPU splitting with --tensor-split.

make clean && make GGML_METAL=1 llama-cli -m model.gguf -ngl 99 -p "Hello" make clean && make GGML_CUDA=1 llama-cli -m model.gguf -ngl 35 -p "Hello" make LLAMA_HIP=1 llama-cli -m model.gguf -ngl 999 llama-cli -m model.gguf -t 8 -p "Hello"

Safety notes: scripts and network access

This skill runs scripts and uses the network. It contacts huggingface.co to search and read model files, and github.com for llama.cpp. The conversion step runs convert_hf_to_gguf.py with Python, and it downloads source weights with the hf CLI before converting them. Review what the agent proposes to run, especially downloads, builds and conversions, before you approve them.

Authentication is manual. The skill does not handle your token for you, so set up hf auth login yourself if you need gated or private models.

  • Downloads can be large. Check repo size before pulling a model.
  • Conversion needs Python and the convert_hf_to_gguf.py script, and is only needed when no GGUF exists.
  • The server is intended for local use; the example uses a placeholder key of no-key.

What people use it for

  • Running a privacy-sensitive chat or coding assistant entirely on your own laptop.
  • Serving a local OpenAI-compatible endpoint so an app you are building can call a model without a cloud API.
  • Finding a GGUF that fits a 16 GB Mac or an 8 GB graphics card and choosing its quant.
  • Converting a Transformers model that has no GGUF release into a quantized GGUF file.
  • Pinning an exact GGUF file so a server always loads the same model.

Frequently asked questions

Is the Hugging Face Local Models skill free?

The skill is published under the Apache-2.0 licence, as listed for huggingface/skills. The models you download and the tools it relies on, such as llama.cpp, each have their own licences, so check those before commercial use.

What do I need before using this skill?

You need llama.cpp installed with llama-cli and llama-server available, and comfort with the terminal. Gated repos need the Hugging Face CLI and an account token set up with hf auth login. Conversion additionally needs Python with convert_hf_to_gguf.py, and the skill needs internet access to huggingface.co and github.com.

Where do I install an agent skill like this one?

Unzip the skill into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder for one project. You can also upload it in the Claude apps under Settings, Capabilities, Skills.

Why does the skill need llama.cpp instead of a one-click app?

The skill generates and runs llama.cpp commands rather than providing its own runtime. Installing llama.cpp through Homebrew, winget, or a source build is a prerequisite, so this is not a one-click setup.

What is the difference between a GGUF file and a quant like Q4_K_M?

GGUF is the file format llama.cpp loads. A quant such as Q4_K_M is a compression level applied to that model, with smaller quants using less memory at some cost in accuracy. The skill helps you pick one that fits your RAM or VRAM.

Can I use this skill with gated or private models?

Yes, but authentication must be set up manually. Run hf auth login with your account token and make sure you have been granted access to the repo.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits developers and technical users who want to run open models privately on their own machine or serve them to an app through a local OpenAI-compatible endpoint, especially when RAM or VRAM is tight. It is not a good fit if you need the most capable cloud models, want a one-click setup, or are not comfortable with the terminal. Local models on consumer hardware are usually smaller and less accurate, and the skill does not change that.

Download the skill