Skip to content
Get packs

LoRA

Python package loralib for adapting large PyTorch language models with small trainable low-rank matrices.

GitHub Download source
Stars
13.9k
Forks
935
Licence
MIT
Language
Python
Last update
1.9 years ago
Type
Library

No commits in over a year — check for forks or alternatives before adopting.

What it is

This is the code for loralib, a Python package that implements LoRA (Low-Rank Adaptation), a method from a Microsoft research paper. Instead of retraining all of a large language model's weights, LoRA freezes the original weights and learns small pairs of low-rank matrices, so only a tiny fraction of parameters is trained and stored for each task. The README reports results comparable or superior to full fine-tuning on GLUE benchmarks with RoBERTa and DeBERTa, and on GPT-2 for E2E, DART and WebNLG tasks.

  • Replace selected nn.Linear, nn.Embedding or nn.Conv2d layers with loralib equivalents to add low-rank adaptation
  • Freeze the base model and train only the LoRA parameters for a downstream task
  • Save and load only the LoRA parameters as small checkpoints for task switching
  • Reproduce the GLUE experiments using the instructions in examples/NLU/
  • Reproduce the GPT-2 E2E, DART and WebNLG experiments using examples/NLG/

Great for

  • Machine learning engineers adapting RoBERTa, DeBERTa or GPT-2 style models in PyTorch
  • Researchers reproducing the LoRA paper's GLUE or GPT-2 experiments
  • Teams that need to store many small task-specific model variants instead of full copies
  • Developers who already use Hugging Face models and want a parameter-efficient fine-tuning option

Not ideal for

  • People who do not use PyTorch, since the README says only PyTorch is supported
  • Users wanting a ready-made chat app or end-user tool, since this is a library of code, not an application
  • Beginners without experience training neural networks, since the setup requires editing model code
Step-by-step guide: how to use LoRAInstall, first run, examples, tips and FAQs
LoRA alternativesLoRA vs Nano-vLLMLoRA vs QLoRA

How to get it

  1. 1

    Install with a package manager

    You'll need: Python · PyTorch

    Install loralib from PyPI

    pip install loralib

    Alternatively, install from the GitHub repo

    pip install git+https://github.com/microsoft/LoRA
  2. 2

    Run it

    • Install loralib with pip install loralib.
    • Replace a supported layer in your model, for example lora.Linear(in_features, out_features, r=16).
    • Call lora.mark_only_lora_as_trainable(model) before the training loop begins so only LoRA parameters train.
    • Train as usual, then save only the LoRA state dict for a small checkpoint.
  3. 3

    Get the most out of it

    • Use the examples/NLU/ and examples/NLG/ directories as working references for integrating with RoBERTa, DeBERTa and GPT-2
    • Download the released LoRA checkpoints for GLUE and GPT-2 tasks to compare against your own results (the base pre-trained model is still required)
    • Check the Hugging Face PEFT library, which the README says supports LoRA, if you want a more maintained integration path
    • Use the MergedLinear layer when a single nn.Linear holds multiple projections, such as a combined attention qkv layer

Downloads

Source code (.zip)main branch · straight from GitHubOfficial website & docsarxiv.org/abs/2106.09685

Strengths

  • Trains and stores far fewer parameters than full fine-tuning, according to the README's comparison tables
  • Adds no inference latency, per the README, because the low-rank matrices can be merged into the original weights
  • Includes working example code for GPT-2, RoBERTa and DeBERTa, and is supported by Hugging Face's PEFT library

Trade-offs

  • Only supports PyTorch and a limited set of layer types (nn.Linear, nn.Embedding, nn.Conv2d) according to the README
  • Benchmark numbers come from the authors' own experiments and paper; results on other models or tasks are not shown in the README
  • The README is truncated and the repo has no latest release, so setup details beyond the basic pip install are unclear from the text provided

How it compares

ProjectStarsTypeLevelUpdatedBest for
LoRA (this)13.9kLibraryintermediate1.9 years agoMachine learning engineers adapting RoBERTa, DeBERTa or GPT-2 style models in PyTorch
qlora11.1kLibraryadvanced2.4 years agoResearchers reproducing or extending the QLoRA paper's finetuning results
nano-vllm16.3kLibraryintermediate6 months agoDevelopers who want to understand how LLM inference engines work internally
nni14.4kLibraryintermediate2.3 years agoResearchers comparing hyperparameter tuning algorithms such as TPE, Hyperband, or Bayesian methods
pytorch_geometric24.1kLibraryintermediate15 days agoResearchers who want ready-made implementations of published graph neural network methods
onnxruntime22.2kLibraryintermediate4 days agoDevelopers who want to deploy a trained model inside an application or service

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.