Skip to content
Get packs

Guide · Agent skill

How to use the GPTQ 4-Bit Quantization agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The GPTQ agent skill teaches your AI agent how to compress large language models to 4-bit precision using post-training quantization, so they fit on consumer NVIDIA GPUs. It is written for developers and ML engineers who want working code for loading pre-quantized models, quantizing their own, and choosing settings. After this guide you will know what the skill does, how to install it, how to ask for it in plain language, and what hardware and software it needs.

GPTQ 4-Bit Model QuantizationOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the GPTQ skill ZIP from this page.
  2. 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for personal use or .claude/skills/ inside a project.
  3. 3Check your hardware and Python setup. Confirm you have an NVIDIA GPU with CUDA and Python with the required libraries installed.
  4. 4Ask for what you need. Describe the model, GPU and goal, such as memory, speed or accuracy, in a plain request that mentions GPTQ.
  5. 5Review and run the code. Read the generated script, check model licences and your token settings, then run it.
  6. 6Validate results. Test the quantized model on your own tasks, since accuracy depends on the model and settings.

What is an agent skill?

An agent skill is a folder containing a SKILL.md file and optional reference documents that give an AI agent specific instructions for a task. When a request matches the skill's description, the agent reads those instructions and follows them while it works.

What the GPTQ skill makes the agent do

The GPTQ skill covers post-training 4-bit quantization of large language models with group-wise quantization. The agent can write code that loads pre-compressed GPTQ models, quantizes your own model using calibration data, and select bit width, group size and kernel backend based on your accuracy, memory and speed needs. It also explains when GPTQ is a better fit than AWQ or bitsandbytes.

The source states that 4-bit GPTQ gives about 4× memory reduction, with roughly 1.5% perplexity increase for the standard setting, and 3-4× faster inference than FP16. These figures depend on the model and settings, so validate results on your own tasks.

  • Load pre-quantized models from Hugging Face with auto-gptq
  • Quantize your own model with a calibration dataset such as C4
  • Choose between ExLlamaV2, Marlin and Triton kernel backends
  • Set up QLoRA fine-tuning with transformers and PEFT

Install the skill

Download the ZIP for this skill from this page. Then either unzip it into your personal skills folder at ~/.claude/skills/, or into a project's .claude/skills/ folder to make it available only in that project.

You can also upload the skill in the Claude apps under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/
  • Project only: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills

Set up your environment

The code the skill writes needs Python, the auto-gptq package, and the transformers, accelerate, optimum, peft and datasets libraries. Quantization and fast kernels need an NVIDIA GPU with CUDA. Marlin needs Ampere or newer GPUs, and Triton needs Linux. Some models, such as meta-llama models, require accepting a licence on Hugging Face, and pushing models to the Hub requires a Hugging Face account and token.

The skill's source shows these install commands for auto-gptq. Check the official package documentation before running them on your machine.

  • Python with auto-gptq, transformers, accelerate, optimum, peft and datasets
  • NVIDIA GPU with CUDA; Marlin needs Ampere or newer
  • Linux for Triton
  • Hugging Face account and token if you push to the Hub
pip install auto-gptq pip install auto-gptq[triton] pip install auto-gptq --no-build-isolation pip install auto-gptq transformers accelerate

Trigger the skill with natural requests

You do not need special syntax. Describe the outcome you want and mention GPTQ or 4-bit quantization. The agent uses the skill's description to decide when to apply it.

Include the model name, your GPU, and whether you want speed, memory savings or accuracy, because the skill picks settings based on those needs.

  • "Load TheBloke/Llama-2-13B-Chat-GPTQ on my RTX 4090 with the fastest backend."
  • "Quantize my 7B model to 4-bit GPTQ with group size 128 and save it locally."
  • "Set up QLoRA fine-tuning on a GPTQ model using LoRA on q_proj and v_proj."
  • "Compare GPTQ and AWQ for my 70B deployment and tell me which fits my GPU."

Choose settings for your goal

The source recommends group size 128 as the default for 4-bit. Smaller groups favour accuracy but cost speed, and 3-bit trades accuracy for more compression. Calibration data affects quality: the guide recommends 128-256 samples of 512 tokens each.

  • Standard 4-bit, group size 128: recommended default
  • Maximum accuracy: 4-bit with group size 32 and desc_act=True
  • High compression: 3-bit, which the source says adds about 3% perplexity increase

Safety notes: scripts and network access

This skill runs scripts and uses the network. The code it writes downloads models from Hugging Face, can load calibration data with datasets, and can push models to the Hub with your token. Review any generated script before running it, keep your Hugging Face token private, and check the licence of any model you download.

The skill cannot provide hardware. If your machine has no NVIDIA GPU, the generated code will not give you the speed or memory benefits described.

  • Review scripts before running them
  • Keep your Hugging Face token out of shared files
  • Accept model licences on Hugging Face before downloading gated models

What people use it for

  • Running a 70B-class model on a consumer NVIDIA GPU using a pre-quantized GPTQ checkpoint
  • Quantizing your own fine-tuned model with calibration data and pushing it to the Hugging Face Hub
  • Fine-tuning a GPTQ model with LoRA adapters through PEFT (QLoRA)
  • Comparing GPTQ with AWQ or bitsandbytes for a specific GPU and accuracy target

Frequently asked questions

Is the GPTQ skill free?

The skill is published under the MIT licence, as stated on this page. The libraries it uses, such as auto-gptq and transformers, have their own licences, and models you download may have licences or access terms of their own.

What do I need to use the GPTQ skill?

You need Python, the auto-gptq, transformers, accelerate, optimum, peft and datasets libraries, and an NVIDIA GPU with CUDA. Marlin needs Ampere or newer GPUs and Triton needs Linux. Some models also need a Hugging Face account, token, or accepted licence.

Does the skill work without an NVIDIA GPU?

The source describes CUDA-based code, and the skill cannot provide hardware. Without an NVIDIA GPU the generated code will not deliver the memory and speed benefits described.

How much accuracy do I lose with GPTQ?

The source cites about 1.5% perplexity increase for the standard 4-bit setting and about 3% for 3-bit. These figures depend on the model and settings, so test the quantized model on your own tasks.

How does GPTQ compare with AWQ or bitsandbytes?

According to the skill, AWQ may give slightly better accuracy and supports Marlin on newer GPUs, while bitsandbytes is simpler to integrate and supports 8-bit. GPTQ is suited to fitting large models on limited memory and to pre-quantized model files.

Why does my GPTQ install fail or run slowly?

The skill's troubleshooting reference covers CUDA version mismatches, build errors that may need --no-build-isolation, and slow inference, which it suggests addressing by trying a different backend such as ExLlama, Marlin or Triton.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are an ML engineer or developer with an NVIDIA GPU who needs to run or fine-tune large models in 4-bit and is comfortable with Python. If you have no GPU, limited Python experience, or want a one-click tool, pick a finished application or a simpler quantization route instead.

Download the skill