Guide · Agent skill
How to use the SGLang Inference Serving agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The sglang skill is a reference guide that teaches your AI agent how to set up and use SGLang, an open-source framework for running large language models as a server. Once it is installed, the agent can write code that launches an SGLang server, forces model output into formats such as JSON, regex patterns or grammars, and reuses cached computation for repeated prompts. This guide covers what the skill does, how to install it, how to ask for it, and what you need before it is useful.
SGLang Inference ServingOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Check your environment. Confirm you have Python with pip and an NVIDIA GPU with CUDA. Without a GPU this skill is of limited use.
- 2Install the skill. Download the ZIP from this page, unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Install SGLang. Run pip install "sglang[all]" in your Python environment, then add torch and transformers if they are missing.
- 4Ask for what you need. Describe the task in plain language, such as wanting JSON output from a model served by SGLang.
- 5Launch the server. Have the agent run python -m sglang.launch_server with your model path and port 30000, or adjust the port to suit.
- 6Test a constrained output. Run a small structured generation, such as an email regex or a JSON schema, and check the output before relying on it.
What is an agent skill and what does the sglang skill do?
An agent skill is a folder of instructions that an AI agent loads when a task matches it. The sglang skill is built around a SKILL.md file and three reference documents on deployment, RadixAttention and structured generation.
The skill does not do the work on its own. It gives the agent the knowledge to write SGLang code for you: launching a server, running structured generation, and designing prompts so that shared parts are cached.
- Launch an SGLang server for a model such as Meta-Llama-3-8B-Instruct
- Force output into JSON (with a schema), regex patterns or an EBNF-style grammar
- Structure agent prompts so that system prompts and tool definitions are reused via RadixAttention
- Choose between SGLang, vLLM and TensorRT-LLM for a given workload
How do you install the sglang skill?
Download the ZIP for the skill from this page and unzip it. For a personal setup, place the unzipped folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.
If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload
What do you need before you start?
The skill assumes a technical setup. You need a Python environment with pip, and for practical use an NVIDIA GPU with CUDA, because the flashinfer wheels are CUDA-specific. The skill's dependencies are the sglang package (installed as sglang[all]), plus torch and transformers.
Model weights such as meta-llama models on Hugging Face may require an account and acceptance of the license before you can download them.
- Python environment and pip
- NVIDIA GPU with CUDA for practical use
- Packages: sglang[all], torch, transformers
- Hugging Face access to gated model weights, if you use them
pip install "sglang[all]"How do you start an SGLang server?
The skill's basic launch command starts a server for Llama 3-8B on port 30000. Enabling RadixAttention is listed as the default in the example comment. For models too large for one GPU, the skill shows tensor parallelism with the --tp flag.
- Basic server: --model-path and --port
- Prefix caching: --enable-radix-cache
- Multiple GPUs: --tp 4 for a 70B model
python -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3-8B-Instruct \
--port 30000How do you get structured JSON or regex output?
Structured generation is the main reason to use this skill. You write a function that builds a prompt, then call sgl.gen with a constraint. The skill shows a regex constraint for a small JSON object and a json_schema constraint for a fuller schema with an enum field.
Regex constraints are simple to read but only match the pattern you give them, so keep the pattern specific to the output you need.
- regex= constrains output to a pattern (for example an email address)
- json_schema= constrains output to a JSON schema
- grammar= accepts an EBNF-style grammar
@sgl.function
def extract_email(s, text):
s += f"Extract email from: {text}\n"
s += "Email: "
s += sgl.gen("email", max_tokens=50, regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')How does prefix caching help agents and multi-turn apps?
RadixAttention caches and reuses the computation for prompt text that repeats across requests, such as a system prompt or a list of tools. The skill's example shows the first request computing the full prompt and later requests reusing the shared system prompt. The skill describes this as automatic, with zero configuration.
The speedup figures in the skill (5×, 10×, 3×) come from the source's own benchmarks and depend heavily on workload. Treat them as indications, not guarantees, and measure on your own prompts.
- Keep system prompts and tool definitions identical across requests so the prefix matches
- Put the changing user message at the end of the prompt
- Use multi-turn conversations with shared context
What people use it for
- Extracting structured fields such as name, age and occupation from free text as JSON
- Pulling email addresses or other patterned values out of documents with a regex constraint
- Building an agent with a long system prompt and tool list that is reused across many calls
- Serving an open-weight model on your own GPU server behind an API on port 30000
- Comparing SGLang with vLLM for a few-shot or agent workload on your own hardware
Frequently asked questions
How do I install the sglang agent skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps, you can instead upload it under Settings → Capabilities → Skills.
Is the sglang skill free and what is its licence?
The skill is listed under the MIT licence. SGLang itself is an open-source framework installed with pip. Model weights you download may have their own licence terms and may require accepting a license on Hugging Face.
What hardware do I need to use SGLang?
The skill recommends an NVIDIA GPU with CUDA for practical use, since flashinfer wheels are CUDA-specific. The framework is aimed at running models on server hardware, so a laptop without a GPU is not a good fit.
When should I use SGLang instead of vLLM?
The skill suggests SGLang for structured outputs, agent workflows and repeated prefixes. It suggests vLLM for simple text generation without structure, or when you want a mature, widely tested production system.
Why does my structured output not match what I expected?
Constraints only match the pattern or schema you supply, so check that your regex or JSON schema describes exactly the output you want. Test with a few prompts before relying on it, and confirm that the model name and port in your server match your client code.
Does the skill run scripts or use the network?
The skill is listed as not running scripts, but it does use the network. The SGLang server and the model downloads from Hugging Face both involve network access, so review what the agent runs and which models it fetches.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits engineers with an NVIDIA GPU who need reliable JSON or regex output, or who build agents with long, repeated system prompts and tool lists. If you only need simple text generation or have no GPU, vLLM or a hosted API will be simpler, and you should not count on the headline speed figures as guarantees.
Download the skill