Guide · Agent skill
How to use the Knowledge Distillation for LLMs agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The knowledge-distillation skill gives an AI agent a guided playbook for knowledge distillation, where a large teacher model is used to train a smaller student model that keeps much of the teacher's behaviour. After this guide you will know what the skill covers, how to install it from the ZIP or the Claude apps, how to ask for it in plain language, and what hardware and software you need before trying the code it produces.
Knowledge Distillation for LLMsOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download the ZIP. Get the knowledge-distillation ZIP from the directory page. It contains SKILL.md and references/minillm.md.
- 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder.
- 3Or upload it in the Claude apps. In the Claude apps, go to Settings, then Capabilities, then Skills, and upload the ZIP.
- 4Prepare your environment. Install Python, PyTorch, transformers, datasets and accelerate, and make sure you have access to the teacher and student weights and a GPU for large teachers.
- 5Ask for what you need. Describe your teacher, student, goal and constraints in plain language, then review the code the agent produces before running it.
What does the knowledge-distillation skill make the agent do?
Once the skill is loaded, the agent works from a guided playbook on knowledge distillation for large language models. It can write PyTorch and Hugging Face code for distillation losses, including temperature scaling, soft targets, reverse KL as used in MiniLLM, and logit distillation. It can also sketch training loops and generate synthetic training data from a teacher model.
The skill is aimed at engineers who want faster, cheaper inference from models they control. It is a source of starter code and explanations, not a finished training pipeline.
- Compress a large model (the source gives 70B to 7B as an example) into a smaller one.
- Write a combined hard-label and soft-label loss with a temperature setting.
- Explain forward versus reverse KL divergence for generative models.
- Draft response distillation: generate answers from a teacher, then fine-tune a student on them.
How do I install the knowledge-distillation skill?
Option one: download the ZIP from the directory page and unzip it into your personal skills folder, ~/.claude/skills/, or into a project's .claude/skills/ folder so it applies only to that project.
Option two: in the Claude apps, open Settings, go to Capabilities, then Skills, and upload the ZIP there.
The skill folder includes SKILL.md (the main playbook) and references/minillm.md (a detailed note on MiniLLM and reverse KL). The skill itself is MIT licensed.
How do I trigger the skill with natural requests?
You do not need a special command. Describe what you want in ordinary language and mention distillation, a teacher and student model, or model compression. The skill's own description lists the situations it is meant for, so your request should match one of them.
Good requests name the goal, the models and the constraint. Vague requests such as 'make my model smaller' will get more generic output.
- Write a PyTorch loss that distils a 70B teacher into a 7B student with temperature 2 and alpha 0.7.
- Show me reverse KL distillation in the MiniLLM style for my generative model.
- I want to reduce inference costs by training a smaller model on responses from my teacher model. Give me the training loop.
- Explain when to use forward KL versus reverse KL for text generation.
What do I need before running the code?
The skill assumes you already work with Python and PyTorch. The code needs the torch, transformers and datasets libraries. accelerate, deepspeed and wandb are needed for training and are optional for experiments. Practical training of large teachers needs GPU hardware.
The examples use gated Llama-2 models on Hugging Face, so you need access to both teacher and student weights. The MiniLLM implementation is optional and lives in the microsoft/LMOps repository.
- Python with PyTorch (torch)
- Hugging Face transformers and datasets, plus accelerate
- deepspeed and wandb (optional, for training)
- Access to teacher and student model weights
- GPU hardware for large teachers
- Optional: microsoft/LMOps for MiniLLM
pip install transformers datasets accelerate
pip install torch deepspeed wandb
git clone https://github.com/microsoft/LMOps
cd LMOps/minillm
pip install -e .How does the reverse KL (MiniLLM) example work?
The skill contrasts two KL directions. Forward KL, the standard choice, has the student match the teacher's average behaviour and focus on its most likely outputs. Reverse KL, used in MiniLLM, asks the student to cover the teacher's range of behaviours, which the source says suits diverse text generation.
The skill's reverse_kl_loss function is a short sketch: it takes softmax probabilities from the teacher, log-softmax from the student, and returns a weighted sum scaled by temperature squared. Treat it as a starting point and check it against the MiniLLM repository before relying on it.
def reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):
p_teacher = F.softmax(teacher_logits / temperature, dim=-1)
log_p_student = F.log_softmax(student_logits / temperature, dim=-1)
reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()
return reverse_kl * (temperature ** 2)What safety and licensing points should I know?
The skill runs scripts and uses the network. Its code downloads model weights from Hugging Face and may clone the LMOps repository. Read any code before running it, and run training in an environment you control.
Do not use this skill to copy outputs from a proprietary model's API to train a competing model. The skill does not check licences, and provider terms may forbid this. Check the terms of any teacher you use.
The skill's '90%+ performance' figure is a general claim, not a verified result. Measure your own student against your own tests before deploying it.
- Review scripts and code before running them.
- Confirm you have the right to use each teacher model and its outputs.
- Benchmark the student on your own task.
What tips help the most?
Start with the basic logit-based loss and a small student before moving to reverse KL. The skill's guidance is to use temperature between 2 and 5, with 2 a common default. Higher temperature exposes more of the teacher's ranking information.
Alpha controls the mix of teacher and label signals. The skill lists 0.5 for balance, 0.7 for more weight on the teacher, and 0.3 for more weight on the labels.
- Try temperature 2 first, then test values up to 5.
- Use alpha 0.5 as a neutral start, then adjust.
- Use the response distillation approach when you want to generate training data from a teacher.
- Ask the agent to explain each line of the loss before you run it.
What people use it for
- Compressing a large teacher model into a smaller student to cut inference costs.
- Writing a combined hard-label and soft-label distillation loss with a chosen temperature and alpha.
- Experimenting with reverse KL distillation for a generative model in the MiniLLM style.
- Generating synthetic responses from a teacher model and fine-tuning a student on them.
Frequently asked questions
Is the knowledge-distillation skill free to use?
The skill is published under the MIT licence, as stated in its SKILL.md. The licence covers the skill files, but you still need to check the terms of any teacher model or API you use for distillation.
How do I install the knowledge-distillation agent skill?
Download the ZIP from the directory page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. Alternatively, in the Claude apps, go to Settings, then Capabilities, then Skills, and upload the ZIP.
What do I need to run the knowledge distillation code?
You need Python, PyTorch, Hugging Face transformers and datasets, and accelerate. Training also uses deepspeed and wandb, which are optional. Large teachers need GPU hardware, and the examples need access to gated Llama-2 weights.
Can a non-expert use this skill?
Not easily. The skill assumes familiarity with PyTorch and substantial compute. Without GPUs or experience with PyTorch, you will struggle to run the code it produces.
Does this skill guarantee the student keeps the teacher's performance?
No. The '90%+ performance' figure in the skill is a general claim rather than a verified result. Test your student on your own task before relying on it.
How does knowledge distillation compare with simply fine-tuning a small model?
Distillation trains the student to match the teacher's output distribution, not only the true labels, which is why the skill combines soft targets with hard-label cross-entropy. Response distillation is closer to standard fine-tuning on teacher-generated text. Which fits better depends on your data and goals.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you are an ML engineer or researcher who controls your models, has GPU access and wants starter code for logit-based or MiniLLM-style distillation. Pick something else if you want a turnkey tool, lack PyTorch experience, or plan to distil from a proprietary API whose terms may forbid it.
Download the skill