Skip to content
Get packs

Quantizing Models with bitsandbytes

Shrink large language models to 8-bit or 4-bit so they fit on smaller GPUs, and train with QLoRA

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

This skill teaches an AI agent how to use the bitsandbytes library to compress large language models so they need much less GPU memory. Once installed, the agent can write code that loads a model in 8-bit or 4-bit precision, pick sensible settings for a given GPU size, fine-tune with QLoRA (training small adapters on a compressed model), and use 8-bit optimizers to save training memory. The skill is a set of instructions and code templates with reference notes on memory and quantization formats; it does not itself run anything.

Great for

  • ML engineers who want to load a 7B to 70B model on a consumer or limited GPU
  • Developers fine-tuning an existing Hugging Face model on a single GPU with QLoRA
  • People who need a quick checklist for choosing between 8-bit and 4-bit settings

Skip it if

  • Users without an NVIDIA GPU or with no ML environment, since the setup depends on Python, PyTorch and CUDA-style GPU tooling; the agent cannot provide that hardware
  • Production deployments needing guaranteed accuracy, because the skill claims under 1% accuracy loss without benchmarks in the files, so results must be validated on your own tasks
  • Non-technical users looking for a finished app, since this produces code for the user to run rather than a ready tool

Try asking

  • “Help me load Llama 2 7B in 4-bit on my 12GB GPU and check how much memory it uses.”
  • “Write a QLoRA fine-tuning script for my Mistral model using LoRA adapters on q_proj and v_proj.”
  • “Replace AdamW with an 8-bit optimizer in my Hugging Face Trainer setup to reduce training memory.”
  • “Estimate how much VRAM I need to run a 13B model at INT8 versus INT4.”
Step-by-step guide: how to use Quantizing Models with bitsandbytesInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    bitsandbytes.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip bitsandbytes.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip bitsandbytes.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • State your GPU model and VRAM up front so the agent can pick 4-bit or 8-bit appropriately.
  • Test the quantized model on a few of your own prompts and compare outputs with the full-precision version before relying on it.
  • Ask the agent to print torch.cuda.memory_allocated() after loading to confirm the real memory savings.

What's inside

  • references/memory-optimization.md
  • references/qlora-training.md
  • references/quantization-formats.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.