Skip to content
Get packs

Guide · Agent skill

How to use the Speculative Decoding skill for faster LLMs

Updated · written with AI from the project's own documentation and checked against our catalogue data

The speculative-decoding skill is a set of instruction files that teaches an AI agent how to make large language models generate text faster, using draft models, Medusa multiple decoding heads and lookahead decoding. It is aimed at ML engineers and developers who run or serve models and need lower latency or higher throughput. After this guide you will know what the skill does, how to install it, how to trigger it with plain requests, and what it needs before you rely on it.

Speculative Decoding for Faster LLMsOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the ZIP file for the speculative-decoding skill from our page.
  2. 2Unzip it into a skills folder. Unzip the skill into ~/.claude/skills/ for a personal install, or into a project's .claude/skills/ folder.
  3. 3Or upload it in the Claude apps. In the Claude apps, open Settings, go to Capabilities, choose Skills and upload the ZIP.
  4. 4Prepare your environment. Install Python, transformers, accelerate and torch, and get access to any Llama-2 or Medusa weights you plan to use.
  5. 5Ask for the technique you need. Describe your latency or throughput problem in plain words and name the model or technique if you know it.

What does the speculative decoding skill make the agent do?

Speculative decoding is a family of techniques that speed up text generation without changing the model's architecture. A small or added predictor guesses several upcoming tokens at once, and the main model checks those guesses in a single pass. The skill's documentation says the draft-model approach has zero quality loss, since it is mathematically equivalent to the target model, and that Medusa-1 is lossless.

With this skill loaded, the agent can write code for loading draft models, Medusa multiple-head models or lookahead decoding, and explain the speed and latency trade-offs of each.

  • Draft model: a small model proposes tokens and a large model verifies them.
  • Medusa: extra prediction heads on an existing model predict future tokens without a separate draft model.
  • Lookahead decoding: reformulates generation as equation solving with Jacobi iteration, without draft models or training.

How do you install the speculative decoding skill?

Download the ZIP file for this skill from our page. Then install it in one of two ways. For a personal install, unzip it into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder.

You can also upload the skill in the Claude apps. Open Settings, go to Capabilities, then choose Skills and upload the ZIP there.

  • Personal: ~/.claude/skills/
  • Project: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills

How do you trigger the skill with natural requests?

You do not need a special command. Describe the problem you are solving, such as slow responses, latency for a real-time application, or running a model on limited compute. The skill's description covers inference speed, latency reduction, limited-compute deployment, draft models, tree-based attention, Jacobi iteration and production deployment, so requests that use these ideas are the most likely to load it.

Name the technique if you know it. Naming the model you are using also helps, because the agent can then tailor the code to it.

  • Help me speed up inference for my chatbot. I want to use a small draft model.
  • Write Medusa code to generate faster from a 7B model on one GPU.
  • Explain the latency trade-offs of lookahead decoding compared with draft-model decoding.
  • Set up speculative decoding for my code assistant without changing its answers.

What do you need before you start?

The skill's install notes list Python, the transformers and accelerate packages, and torch (PyTorch) as the core requirements. The Medusa and LookaheadDecoding repositories, cloned from GitHub, are optional. So are vLLM and GPU hardware, which the skill says are needed for practical use.

Access to model weights matters too. The examples use Llama-2 and Medusa models hosted on Hugging Face, so you need access to those weights.

  • Python with transformers, accelerate and torch installed
  • Optional: Medusa and LookaheadDecoding repositories cloned from GitHub
  • Optional: vLLM
  • GPU hardware for practical use
  • Access to Llama-2 and Medusa model weights on Hugging Face
pip install transformers accelerate

What do the code examples cover?

The skill's SKILL.md file has three quick-start examples: a basic draft-model setup using assisted generation in transformers, a Medusa example that calls medusa_generate, and a lookahead decoding example. The draft-model example pairs a large target model with a smaller draft model. The Medusa example uses a pre-trained Medusa model. The lookahead example sets window size, n-gram size and guess size.

The examples use placeholder model names and API details that may differ across library versions, so check them against the versions you have installed before you run them in production.

  • Draft model: target_model.generate with assistant_model set to the draft model
  • Medusa: MedusaModel loading followed by medusa_generate
  • Lookahead: LookaheadDecoding with window_size, ngram_size and guess_size
outputs = target_model.generate(**inputs, assistant_model=draft_model, max_new_tokens=256, do_sample=True, temperature=0.7)

What are the safety notes for scripts and network access?

The skill does not run scripts. Its files are documentation-style instructions plus two reference notes, lookahead.md and medusa.md, and they contain no executable scripts of their own.

The skill does use the network. Its install steps include cloning GitHub repositories and pip installing packages, and its examples load model weights from Hugging Face. Review any code the agent writes before running it, and check which repositories and models it downloads.

  • Runs scripts: no
  • Uses network: yes (GitHub clones, pip installs, Hugging Face model downloads)
  • Review generated code and downloaded models before running them

What tips help you get real results?

Treat the claimed 1.5-3.6× speedup as a range to test, not a guarantee. The gain depends on the model, the quality of the draft or heads, and the hardware. The skill does not benchmark these results, so measure latency and throughput on your own setup.

Start with the option that fits your constraints. A draft model needs a second model; Medusa needs heads trained for your model; lookahead decoding needs no draft model and no training, according to the reference notes.

  • Benchmark your own model and hardware before deciding.
  • Check placeholder model names and API calls against your installed library versions.
  • If you cannot change inference settings or model code, these techniques may not be available to you.

What people use it for

  • Lowering response time for a chatbot that must feel real-time.
  • Speeding up a code assistant without changing the model's answers, using a draft-model setup.
  • Getting more throughput from an existing model on a limited GPU budget by adding Medusa heads.
  • Comparing draft-model, Medusa and lookahead approaches for a specific deployment.

Frequently asked questions

Is the speculative decoding skill free to use?

The skill is listed under the MIT licence. The SKILL.md file names Orchestra Research as the author and MIT as the license. Check the licence file in the ZIP for the exact terms.

What do I need to run speculative decoding?

The skill's requirements are Python, transformers, accelerate and torch. Medusa and LookaheadDecoding repositories, vLLM and GPU hardware are optional but recommended for practical use. You also need access to the model weights you use, such as Llama-2 and Medusa models on Hugging Face.

Does speculative decoding change the answers a model gives?

The skill's documentation says the draft-model method has zero quality loss because it is mathematically equivalent to the target model, and that Medusa-1 is lossless. Medusa-2 fine-tunes the base model, so its behaviour depends on that training.

How much faster will my model run?

The skill cites 1.5-3.6× speedups, but it does not benchmark them. Results depend on the model, the draft quality and your hardware, so measure your own setup.

Can I use this skill with a hosted chat app?

Probably not. The techniques need access to model code or serving infrastructure, so people who only use hosted chat apps cannot change inference settings.

Why does the example code fail for me?

The examples use placeholder model names and API details that may differ across library versions. Check the current documentation for your installed transformers, Medusa or LookaheadDecoding version and adjust the calls.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits ML engineers and developers who control model code or serving setup and want lower latency or higher throughput on limited hardware. It is a poor fit for people who only use hosted chat apps or who expect guaranteed speedups without testing. Benchmark and review any generated code before using it in production.

Download the skill