vLLM LLM Serving
Lets the agent set up and run high-throughput vLLM servers for large language models.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 5 files
What it does
Once installed, the agent can guide you through deploying a large language model behind an OpenAI-compatible API using vLLM, a popular open-source inference engine. It knows the typical commands for single-GPU and multi-GPU setups, quantized models (AWQ, GPTQ, FP8), offline batch processing, Docker deployment, and Prometheus monitoring, and it can troubleshoot out-of-memory errors. It is a reference guide for the agent rather than a tool that serves models by itself.
Great for
- Engineers deploying production LLM APIs who need high throughput on GPUs
- Teams wanting an OpenAI-compatible endpoint for self-hosted open models
- Developers fitting large models into limited GPU memory using quantization
Skip it if
- Users without NVIDIA or otherwise supported GPU hardware, since the guidance assumes GPU serving; llama.cpp-style CPU inference is listed as an alternative
- Single-user experiments or one-off generation, where the skill itself recommends plain transformers instead
- Anyone expecting the performance numbers (such as 24x throughput or TTFT under 500ms) to hold for their own hardware and workload without measuring them
Try asking
- “Set up a vLLM server for Llama 3 8B on a single GPU with prefix caching and metrics enabled”
- “Show me how to run a 70B model with AWQ quantization across 4 GPUs using tensor parallelism”
- “Write a Python script that runs offline batch inference over a prompts.txt file with vLLM”
- “Help me troubleshoot an out-of-memory error when loading a model in vLLM”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 5 original files, the MIT licence and an attribution note.
vllm.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip vllm.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip vllm.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Measure TTFT and throughput on your own hardware with a load test before trusting the targets listed in the skill.
- Review every vllm serve command before running it, especially --host 0.0.0.0, which exposes the server on all network interfaces without authentication.
- Check the troubleshooting and optimization reference files for issue-specific guidance beyond the main SKILL.md.
What's inside
- references/optimization.md
- references/quantization.md
- references/server-deployment.md
- references/troubleshooting.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.