Knowledge Distillation for LLMs
Teaches the agent to shrink large language models into smaller student models using teacher-student training.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 2 files
What it does
This skill gives an AI agent a guided playbook for knowledge distillation, a technique where a large, capable AI model (the teacher) is used to train a smaller, cheaper model (the student) so the small model keeps much of the teacher's behaviour. Once installed, the agent can write PyTorch and Hugging Face code for distillation losses (temperature scaling, soft targets, reverse KL as in MiniLLM), set up training loops, and generate synthetic training data from a teacher model. It is aimed at engineers who want faster, cheaper inference from models they control.
Great for
- ML engineers who need a smaller model to serve at lower inference cost
- Researchers experimenting with MiniLLM-style reverse KL distillation
- Developers who want starter code for logit-based or response-based distillation
Skip it if
- Anyone wanting to copy outputs from a proprietary model's terms-restricted API to train a competing model, since provider terms may forbid this and the skill does not check licences
- Non-technical users without GPUs or experience with PyTorch, because the code requires substantial compute and setup
- Expecting guaranteed performance retention; the skill's '90%+ performance' figure is a general claim, not a verified result
Try asking
- “Write a PyTorch loss function that distills a 13B teacher into a 1B student using temperature 2 and alpha 0.7”
- “Show me how to generate synthetic responses from a teacher model and fine-tune a smaller student on them”
- “Implement MiniLLM reverse KL loss for my student model and explain when it beats forward KL”
- “Help me choose temperature and alpha values for distilling a classifier-style LLM”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 2 original files, the MIT licence and an attribution note.
knowledge-distillation.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip knowledge-distillation.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip knowledge-distillation.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Check the license and terms of any teacher model before generating training data from it; the skill does not verify this.
- Start with the basic distillation loss and a small teacher-student pair to validate the pipeline before scaling up.
- Review the generated code before running it, since it downloads large models and may use significant GPU time and cloud costs.
What's inside
- references/minillm.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.