Guide · Agent skill
How to use the vLLM LLM Serving agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The vLLM LLM Serving skill (serving-llms-vllm) gives your AI agent a working reference for deploying a large language model behind an OpenAI-compatible API with vLLM, an open-source inference engine. It is written for engineers who need high throughput on GPUs, teams self-hosting open models, and developers fitting big models into limited GPU memory. After this guide you will know how to install the skill, ask for help with natural requests, and what the skill needs before you start.
vLLM LLM ServingOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip. Download the skill ZIP from this page and unzip it so SKILL.md and references/ are in one folder.
- 2Place the skill. Put the folder in ~/.claude/skills/ for personal use or .claude/skills/ inside a project, or upload the ZIP under Settings → Capabilities → Skills.
- 3Check your hardware. Confirm you have a supported GPU with enough VRAM and that Python and pip are available. Install Docker if you want the containerised option.
- 4Ask with a specific request. Describe your model, GPU count and goal, such as an OpenAI-compatible endpoint or fitting a large model in limited memory.
- 5Test under light load. Run the locust load test the skill describes and check time to first token and throughput against your own targets.
- 6Enable monitoring. Ask the agent to set up Prometheus metrics and watch vllm:time_to_first_token_seconds and vllm:gpu_cache_usage_perc.
What is an agent skill, and what does this one do?
An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches the skill's description. This one is built around vLLM, which uses PagedAttention (a block-based KV cache) and continuous batching to serve models with high throughput.
Once installed, the agent can guide you through single-GPU and multi-GPU deployment, quantized models (AWQ, GPTQ, FP8), offline batch processing, Docker deployment, Prometheus monitoring, and troubleshooting out-of-memory errors. It is a reference guide for the agent. It does not serve models by itself.
- Typical commands for serving a model with vllm serve
- Settings such as --gpu-memory-utilization, --max-model-len and --tensor-parallel-size
- Offline batch inference with the Python LLM class
- Docker deployment using the vllm/vllm-openai image
- Prometheus metrics on a metrics port
Install the vLLM skill from the ZIP
Download the ZIP for this skill from this page and unzip it. The folder must contain SKILL.md and the references/ folder. Then place it where your agent looks for skills:
For a personal install, unzip into ~/.claude/skills/. For a single project, unzip into the project's .claude/skills/ folder.
In the Claude apps you can instead upload the ZIP under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/
- Project: .claude/skills/ inside your project folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Trigger the skill with natural requests
You do not need a special command. The skill loads when your request matches its description, so describe the job in plain words. Mention the things the skill covers: an OpenAI-compatible endpoint, quantization, limited GPU memory, throughput or latency.
Good requests name your goal and your hardware. The more specific you are about model size and GPU count, the more useful the answer.
- Help me deploy Llama 3 8B behind an OpenAI-compatible API on one GPU.
- Serve a 70B model on 4 GPUs with AWQ quantization. What flags should I use?
- I get out-of-memory errors when vLLM loads my model. What should I change?
- Write an offline batch script that runs a list of prompts from prompts.txt through vLLM.
Example: start a server and query it
The skill's quick start installs vLLM with pip, starts a server, and queries it with the OpenAI Python SDK. Ask the agent to adapt these commands to your model and port.
Note that the example model meta-llama/Llama-3-8B-Instruct may require an accepted license and a Hugging Face token.
pip install vllm
vllm serve meta-llama/Llama-3-8B-InstructCommon fixes the skill covers
For out-of-memory errors during model loading, the skill suggests lowering --gpu-memory-utilization (for example to 0.7), reducing --max-model-len, or using quantization such as --quantization awq.
For slow first tokens, it suggests --enable-prefix-caching for repeated prompts and --enable-chunked-prefill for long prompts. For low throughput, it suggests raising --max-num-seqs and checking GPU use with nvidia-smi.
- Out of memory: lower --gpu-memory-utilization, lower --max-model-len, or quantize
- Slow first token: --enable-prefix-caching or --enable-chunked-prefill
- Custom models failing to load: --trust-remote-code
- Low throughput: higher --max-num-seqs, then check GPU utilization
Safety notes: network use and scripts
This skill does not run scripts. It does use the network: the guidance involves downloading model weights from Hugging Face, installing Python packages, and running servers that accept requests. Review any command before you run it, especially those that bind to 0.0.0.0, which exposes the server beyond your machine.
The examples also rely on Python packages openai and locust, and on Docker for the containerised option. Check that the model you choose allows your intended use under its own license.
- No scripts are run by the skill itself
- Model downloads and servers use the network
- Using --host 0.0.0.0 makes the endpoint reachable from other machines
- Review model licences before production use
What people use it for
- Deploying a self-hosted open model behind an OpenAI-compatible endpoint for an internal chatbot
- Fitting a 70B model onto fewer GPUs with AWQ quantization
- Running offline batch inference over a large prompt file
- Packaging a vLLM server in Docker for consistent deployment
- Diagnosing out-of-memory errors when a model fails to load
Frequently asked questions
Is the vLLM agent skill free?
The skill is published under the MIT licence, as listed on this page. vLLM itself is installed with pip install vllm, and the model you serve has its own license, which you should check before use.
What do I need to use this skill?
You need Python with vLLM installed, GPU hardware (the guidance assumes NVIDIA-class GPUs with enough VRAM), and Hugging Face access. Docker is needed only for the containerised option, and openai and locust are used in the example snippets.
Where do I put the skill files?
Unzip the ZIP into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. In the Claude apps, upload it under Settings → Capabilities → Skills.
Can I use vLLM without a GPU?
The skill assumes GPU serving and does not cover CPU inference. The skill lists llama.cpp as an alternative for CPU and edge inference.
What should I use instead of vLLM?
The skill suggests llama.cpp for CPU or edge single-user use, HuggingFace transformers for research and one-off generation, TensorRT-LLM for NVIDIA-only maximum performance, and Text-Generation-Inference if you already use the Hugging Face ecosystem.
Will the performance numbers in the skill apply to my setup?
Not necessarily. The skill cites figures such as higher throughput than standard transformers and a TTFT target under 500ms, but these depend on your hardware and workload, so measure them yourself with a load test.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Choose this skill if you are deploying an OpenAI-compatible LLM API on NVIDIA-class GPUs and want a checklist-driven reference for flags, quantization and troubleshooting. Skip it if you have no GPU, want CPU inference, or only need to run a model once for a quick experiment, where the skill itself points to simpler tools.
Download the skill