Guide · Agent skill
How to use the HQQ Model Quantization agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The hqq-quantization skill teaches an AI agent how to shrink large language models by lowering the numerical precision of their weights, so they fit into less GPU memory. It is aimed at developers and engineers who already work with Python and PyTorch and want a fast quantization step. After following this guide you will know what the skill does, how to install it, how to trigger it with plain requests, and what your machine needs before you start.
HQQ Model QuantizationOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ or into your project's .claude/skills/ folder.
- 2Check your hardware. Confirm you have Python, PyTorch 2.0 or newer, and a CUDA GPU. Marlin needs an Ampere or newer GPU.
- 3Install hqq. Run pip install hqq, adding a backend extra such as [marlin] if you plan to use it.
- 4Ask for quantization in plain language. Name HQQ and your target bit width, for example 4-bit, and the model you want to compress.
- 5Provide Hub access if needed. For gated models such as meta-llama, create a HuggingFace account and token with access to the model.
- 6Test the quantized model. Ask the agent to generate a short output from the quantized model to confirm it loads and runs.
What is an agent skill?
An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches the skill's description. The skill for this page is named hqq-quantization and its main instructions are in a file called SKILL.md.
What the skill makes the agent do
The skill covers Half-Quadratic Quantization (HQQ), a method for compressing LLM weights. Its main selling point is that it does not need a calibration dataset, so a model can be quantized without first supplying sample data.
The skill teaches the agent the relevant Python APIs for quantizing models, loading pre-quantized models, choosing backends, and serving results through HuggingFace Transformers or vLLM. It also covers LoRA/PEFT fine-tuning of quantized models.
- Quantize a linear layer or a full model at 8, 4, 3, 2 or 1-bit precision
- Configure group size and per-layer mixed precision
- Pick an inference backend such as pytorch, aten, torchao_int4, marlin or bitblas
- Load or save quantized models with HuggingFace and serve them with vLLM
- Fine-tune a quantized model with LoRA
Install the skill
You can install the skill in one of three ways. Download the ZIP from this page and unzip it into your personal skills folder at ~/.claude/skills/, or unzip it into a project's .claude/skills/ folder to limit it to that project. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
The ZIP contains SKILL.md plus two reference files: references/advanced-usage.md and references/troubleshooting.md.
- Personal: unzip into ~/.claude/skills/
- Project only: unzip into your project's .claude/skills/
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Install the Python packages the skill needs
The skill itself does not run the library for you. Your machine needs the hqq package (version 0.2.0 or newer), PyTorch (2.0.0 or newer) and, for Hub integration, HuggingFace transformers. The source shows the base install command and optional backend extras.
- Base package: hqq
- Extras: torch, torchao, bitblas, marlin (for example hqq[marlin])
- vLLM is needed only for the serving examples
pip install hqq
# With specific backend
pip install hqq[torch]
pip install hqq[torchao]
pip install hqq[bitblas]
pip install hqq[marlin]How to trigger the skill with natural requests
You do not need a special command. Describe what you want in plain language, and the agent will load the skill when your request matches its description. The description mentions quantizing to 4, 3 or 2-bit precision, quantizing without calibration datasets, fast quantization workflows, and deploying with vLLM or HuggingFace Transformers.
Naming the method or the goal helps. Mentioning HQQ directly is the most reliable trigger.
- "Quantize Llama 3.1 8B to 4-bit with HQQ so it fits on my 12 GB GPU."
- "Use HQQ to make a 2-bit version of this model without calibration data."
- "Show me how to serve an HQQ-quantized model with vLLM."
- "Set up LoRA fine-tuning on a model quantized with HQQ."
A basic example
The skill's basic pattern loads a model through HuggingFace with a quantization config. The config sets the bit width and group size. Gated models such as meta-llama need a HuggingFace account and access token, as the source notes.
- nbits sets bits per weight; group_size sets weights per quantization group
- Loading a pre-quantized Hub model skips the quantization step entirely
from transformers import AutoModelForCausalLM, HqqConfig
quantization_config = HqqConfig(nbits=4, group_size=64, axis=1)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
quantization_config=quantization_config,
device_map="auto"
)Choose a backend and troubleshoot common errors
Backend choice depends on your GPU. The skill's guide lists pytorch as the compatible default, aten as a balanced option on CUDA GPUs, and marlin for maximum 4-bit speed, which requires an Ampere or newer GPU. The troubleshooting reference covers missing backend imports, CUDA version mismatches and out-of-memory errors during quantization.
- ModuleNotFoundError for hqq: reinstall with pip install hqq and check with python -c "import hqq"
- ImportError for the marlin backend: install hqq[marlin] or the matching backend package
- CUDA 'no kernel image' errors: check your CUDA and PyTorch versions, then reinstall
What people use it for
- Fitting a large model onto a GPU with limited memory by quantizing it to 4-bit
- Quickly producing a quantized checkpoint without preparing a calibration dataset
- Serving a quantized model through vLLM with the hqq quantization option
- Experimenting with 2-bit or 1-bit quantization on a model you own or have access to
- Fine-tuning a quantized model with LoRA using PEFT
Frequently asked questions
Is the HQQ quantization skill free?
The skill is released under the MIT licence, as stated on this page. The HQQ library and the other packages it uses are separate projects, so check their own licences before commercial use.
What do I need to use the hqq-quantization skill?
You need Python, the hqq package (0.2.0 or newer), and PyTorch (2.0.0 or newer). A CUDA GPU is needed for most optimized backends, and Marlin requires an Ampere or newer GPU. Gated models also need a HuggingFace account and token.
Does HQQ need calibration data?
No. HQQ is designed to quantize models without a sample dataset, which the skill describes as its main advantage. If you have calibration data and need maximum accuracy at low bit-widths, the skill recommends GPTQ or AWQ instead.
Can I use HQQ on a Mac or CPU-only machine?
The skill points CPU-only and Apple Silicon users to llama.cpp with GGUF instead. Most optimized HQQ backends assume a CUDA GPU.
Why does my HQQ install fail with a marlin or CUDA error?
The troubleshooting reference covers missing backend imports, which are fixed by installing the matching extra such as hqq[marlin], and CUDA version mismatches, which are fixed by reinstalling PyTorch with a CUDA build that matches your system and then reinstalling hqq.
Does the skill run code on my machine?
Yes. The skill runs scripts and uses the network, for example to download models from HuggingFace. Review the SKILL.md and reference files before installing, and only grant access to models and accounts you trust.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you are a developer or engineer with Python and a CUDA GPU who needs to shrink a model quickly and may deploy it with HuggingFace Transformers or vLLM. Pick GPTQ or AWQ if you already have calibration data and need the highest accuracy at low bit-widths, and pick llama.cpp with GGUF for CPU-only or Apple Silicon work.
Download the skill