Guide · Agent skill
How to use the AWQ 4-bit quantization agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The awq-quantization skill teaches an AI agent how to compress large language models, roughly 7B to 70B parameters, to 4-bit precision using Activation-aware Weight Quantization (AWQ) so they fit on smaller GPUs and run faster. It is written for ML engineers and developers who deploy open-weight models and need a practical workflow. After following this guide you will know what the skill does, how to add it to your agent, how to phrase requests that trigger it, and what hardware and software it needs.
AWQ 4-bit Model QuantizationOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the awq-quantization ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder.
- 2Check your hardware and software. Confirm you have an NVIDIA GPU with compute capability 7.5+, CUDA 11.8+, and Python 3.8+.
- 3Install autoawq. Run pip install autoawq, adding the kernels or cpu extra only if you need them.
- 4Ask the agent a concrete task. Describe the model, your GPU memory and whether you want to load a pre-quantized model or quantize your own.
- 5Run the generated code. Review the Python the agent produces, then run it to load or quantize the model.
- 6Benchmark and check accuracy. Compare output quality and speed against your baseline, since the skill reports a small perplexity loss that depends on the model.
What is an agent skill and what does this one do?
An agent skill is a folder containing a SKILL.md file with instructions and reference notes that an AI agent loads when a task matches the skill's description. This one is the awq-quantization skill from Orchestra-Research/AI-Research-SKILLs, published under the MIT licence.
With the skill loaded, the agent can explain and carry out AWQ workflows: installing the autoawq library, loading pre-quantized models, quantizing your own model, choosing kernel backends (GEMM, GEMV, Marlin, ExLlamaV2), serving with vLLM, and comparing AWQ with GPTQ and bitsandbytes.
How to install the awq-quantization skill
Download the skill ZIP from this page and unzip it. For a personal setup, place the folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.
You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/awq-quantization/
- Project: .claude/skills/awq-quantization/
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
How to trigger the skill with natural requests
You do not need a special command. Describe the task in plain language that matches the skill's purpose: deploying a large model on limited GPU memory, getting faster inference than GPTQ, or quantizing an instruction-tuned or multimodal model.
Mention the model size, your GPU situation or your serving stack (for example vLLM) so the agent picks the right steps.
- Quantize a 7B chat model to 4-bit with AWQ so it fits on my 12 GB GPU.
- Show me how to load a pre-quantized AWQ model with fused layers and generate text.
- Serve a quantized Llama 2 AWQ model with vLLM.
- Compare AWQ, GPTQ and bitsandbytes for my production chat model.
Examples of what the agent produces
For loading a pre-quantized model, the skill's guidance uses AutoAWQForCausalLM.from_quantized with fuse_layers=True, plus a tokenizer and a generate call. For your own model, it shows a quantize-and-save flow using a quant_config with zero_point, q_group_size 128, w_bit 4 and version GEMM.
Quantization is noted to take roughly 10-15 minutes for a 7B model and about an hour for a 70B model.
pip install autoawqWhat you need before you start
The skill assumes Python 3.8+, CUDA 11.8+ and an NVIDIA GPU with compute capability 7.5 or higher. Marlin kernels require compute capability 8.0+. You also need the autoawq library, transformers 4.45.0 or newer, torch 2.0.0 or newer, and Hugging Face Hub access to download models. vLLM is only needed for vLLM serving.
Optional extras are kernels (optimized CUDA kernels with Flash Attention) and cpu (Intel CPU/XPU optimization).
- Python 3.8+
- CUDA 11.8+ with an NVIDIA GPU of compute capability 7.5+ (8.0+ for Marlin)
- autoawq, transformers>=4.45.0, torch>=2.0.0
- vLLM only for vLLM serving
- Hugging Face Hub access
Safety notes: scripts and network access
According to the skill's listing, it does not run scripts, but it does use the network, mainly to download models from the Hugging Face Hub. Review any code the agent writes before running it, and check the model sources you pass in.
The skill's own deprecation notice says AutoAWQ is officially deprecated and points new projects toward the vLLM llm-compressor project.
Tips for better results
Use the GEMM kernel for batch inference and GEMV for single-token generation, which the skill describes as limited to batch size 1. On Ampere or newer GPUs, the skill describes Marlin as faster on A100 and H100 hardware.
If you hit CUDA out-of-memory errors during quantization, the skill suggests reducing max_calib_samples. For slow inference, it suggests fuse_layers=True. Custom calibration data can improve domain-specific accuracy, and the skill notes that more samples generally help.
- Start with pre-quantized models before quantizing your own.
- Expect roughly 2-4% perplexity degradation, not zero change.
- Fused modules cannot be combined with FlashAttention2, according to the skill.
What people use it for
- Fitting a 7B or 13B instruction-tuned chat model onto a single consumer GPU.
- Serving a quantized model in production with vLLM using quantization set to awq.
- Comparing AWQ, GPTQ and bitsandbytes before choosing a method for a team deployment.
- Running a 70B model across several GPUs with device_map and max_memory settings.
- Quantizing a domain-specific model using your own calibration text.
Frequently asked questions
Is the awq-quantization skill free to use?
The skill is published under the MIT licence, as stated in its SKILL.md. The underlying autoawq library is installed separately with pip, and you should check its own licence before commercial use.
What hardware do I need for AWQ?
The skill requires an NVIDIA GPU with compute capability 7.5 or higher and CUDA 11.8+. Marlin kernels need compute capability 8.0+. Older GPUs such as GTX 10xx and V100 are not supported, according to the troubleshooting notes.
How do I install the autoawq library?
Run pip install autoawq for the default Triton kernels. Use pip install autoawq[kernels] for optimized CUDA kernels with Flash Attention, or pip install autoawq[cpu] for Intel CPU/XPU optimization.
Is AWQ better than GPTQ?
The skill says AWQ generally gives faster inference and better accuracy preservation, citing about 2.5-3x speedup versus roughly 2x for GPTQ. It also notes GPTQ has broader tool ecosystem support and works with ExLlamaV2, so the best choice depends on your stack.
Why does AWQ fail with a CUDA or compute capability error?
The troubleshooting guide lists two common causes: a CUDA version mismatch, which can produce 'no kernel image is available for execution', and a GPU with compute capability below 7.5, which triggers an assertion error. Check your GPU and CUDA version before reinstalling.
Does AWQ work on AMD GPUs?
The skill mentions ExLlamaV2 kernel support for AMD GPUs through version="exllama" in AwqConfig. Its main guidance assumes NVIDIA CUDA hardware.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you are an ML engineer or developer deploying open-weight LLMs on NVIDIA GPUs with limited memory, especially with vLLM. Pick something else if you lack a supported GPU, expect zero accuracy change, or need a no-code tool, since the skill requires running Python.
Download the skill