Guide · Open source
How to use QLoRA: install, finetune a 4-bit model and tips
Updated · written with AI from the project's own documentation and checked against our catalogue data
QLoRA is research code from the paper "QLoRA: Efficient Finetuning of Quantized LLMs" that finetunes large language models while keeping the frozen base model in 4-bit precision and training small LoRA adapter layers on top. This guide is for developers and researchers who have an NVIDIA GPU and want to finetune an open LLM on limited memory. By the end you will know how to set up the repository, run the basic training command, and point it at your own dataset.
QLoRAartidoro/qlora · ★ 11.1k · MIT Install options & downloads Quick start
- 1Check your hardware. Confirm you have an NVIDIA GPU with CUDA, since the README's setup relies on bitsandbytes.
- 2Clone the repository. Run git clone on https://github.com/artidoro/qlora to get the code.
- 3Install PyTorch. Follow the PyTorch instructions for your system before installing the project requirements.
- 4Install requirements. Run pip install -U -r requirements.txt inside the cloned folder.
- 5Run the basic finetune. Start python qlora.py with --model_name_or_path set to your base model.
- 6Use your own dataset. Point --dataset at your data and set --dataset_format if you are not using Alpaca format.
What QLoRA is and who it is for
QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). The README says this reduces memory enough to finetune a 65B parameter model on a single 48GB GPU.
The project is built on bitsandbytes and is integrated with Hugging Face's PEFT and transformers libraries. It is released under the MIT license and is best suited to researchers and developers, not to people looking for a ready-made chat app.
- Good fit: reproducing or extending the QLoRA paper, finetuning an open base model such as LLaMA on your own instruction data.
- Poor fit: non-technical users, anyone without an NVIDIA GPU, and teams that need production-supported software.
Install QLoRA: clone the repo and install requirements
First clone the repository. Then install PyTorch by following the instructions on the PyTorch website. The README notes that you need the latest bitsandbytes library and accelerate and transformers installed from source for 4-bit loading.
Install the project requirements from inside the cloned folder.
- Prerequisites: Python, PyTorch, and an NVIDIA GPU with CUDA for bitsandbytes.
git clone https://github.com/artidoro/qlora
pip install -U -r requirements.txtRun your first finetuning job
The `qlora.py` script is the starting point for finetuning and inference on various datasets. The basic command finetunes a baseline model on the Alpaca dataset. Replace the placeholder with the path or Hugging Face name of your base model.
For models larger than 13B, the README recommends lowering the learning rate.
- Pass `--model_name_or_path` with a local path or model name.
- For models larger than 13B, add `--learning_rate 0.0001`.
- Examples under the `examples/` folder include a generation example with Guanaco at `examples/guanaco_generate.py`.
python qlora.py --model_name_or_path <path_or_name>Configure 4-bit quantization
Quantization is controlled through the BitsandbytesConfig class in transformers. The README lists the options: `load_in_4bit`, `bnb_4bit_compute_dtype`, `bnb_4bit_use_double_quant`, and `bnb_4bit_quant_type`.
The README supports two quantization datatypes, fp4 and nf4, and recommends nf4 because it is theoretically optimal for normally distributed weights. The paged optimizer is enabled with `--optim paged_adamw_32bit`.
- `load_in_4bit`: turns on 4-bit loading.
- `bnb_4bit_compute_dtype`: datatype used for linear layer computations.
- `bnb_4bit_use_double_quant`: turns on nested quantization.
- `bnb_4bit_quant_type`: set to `nf4` as the README recommends.
model = AutoModelForCausalLM.from_pretrained(
model_name_or_path='/name/or/path/to/your/model',
load_in_4bit=True,
device_map='auto',
max_memory=max_memory,
torch_dtype=torch.bfloat16,
quantization_config=BitsandbytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type='nf4'
),
)Train on your own dataset
Set `--dataset` to a path to your own data. If you do not set `--dataset_format`, the script defaults to the Alpaca format. For the self-instruct format, set `--dataset_format` explicitly.
To reproduce Guanaco, select `--dataset oasst1`, which loads the OpenAssistant dataset. The README includes scripts for Guanaco hyperparameters at `./scripts/finetune_guanaco*.sh`. Adjust `per_device_train_batch_size` and `gradient_accumulation_steps` so their product is 16 and training fits your GPUs.
- Alpaca format is the default.
- Self-instruct format needs `--dataset_format="self-instruct"`.
- Multi-GPU training and inference work through Hugging Face's Accelerate, according to the README.
python qlora.py --dataset="path/to/your/dataset" --dataset_format="self-instruct"Evaluate and use the outputs
The repository includes generated outputs for the paper's models for OA and Vicuna queries in the `eval/generations` folder. The README also includes scripts adapted from FastChat that use GPT-4 to evaluate model generations, with scores out of 10 or pairwise comparisons labelled win, loss, or tie.
The README notes that Guanaco is intended for research purposes only and could produce problematic outputs, and that the Guanaco models are intended for uses in line with the LLaMA license.
- Sample generations: `eval/generations`.
- Demo notebooks are linked from the README for hosting a Guanaco Gradio demo in Colab.
What people use it for
- Reproducing the QLoRA paper's finetuning results on a research GPU.
- Finetuning an open LLaMA-family base model on your own instruction dataset.
- Training a Guanaco-style model with the included oasst1 dataset option and scripts.
- Experimenting with 4-bit NF4 quantization settings to see how they affect GPU memory use.
Frequently asked questions
Is QLoRA free to use?
QLoRA's resources in this repository are released under the MIT license, according to the README. The Guanaco models are intended for uses in line with the LLaMA license and require access to the LLaMA models.
What hardware do I need to run QLoRA?
The README relies on CUDA-based bitsandbytes and GPU memory figures, so you need an NVIDIA GPU. The README says a 65B parameter model can be finetuned on a single 48GB GPU.
How do I install QLoRA?
Clone the repository from https://github.com/artidoro/qlora, install PyTorch following its instructions, then run pip install -U -r requirements.txt in the project folder.
Can I finetune QLoRA on my own dataset?
Yes. Pass a path to your dataset with --dataset. The script uses the Alpaca format by default, and you can set --dataset_format="self-instruct" for that format.
What are the alternatives to QLoRA?
The README lists comparable projects including bitsandbytes, PEFT, LLaMA-Factory, and Axolotl. bitsandbytes handles the quantization and PEFT provides the adapter methods that QLoRA builds on.
Is QLoRA suitable for production?
The README frames the work as research and states Guanaco is intended for research purposes only. Teams that need production-ready, supported software should look at other options.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
QLoRA is a good choice for researchers and developers with an NVIDIA GPU who want to finetune large open LLMs on limited memory, using MIT-licensed code that follows the paper. It is not a chat app and is not supported as production software, so non-technical users and teams needing supported tools should choose a different project such as LLaMA-Factory or Axolotl.
Install options & downloads