Guide · Agent skill

How to use the TensorRT-LLM Inference Serving agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The tensorrt-llm skill is an instruction bundle for an AI agent, covering NVIDIA's TensorRT-LLM library for running large language models on NVIDIA GPUs. It is aimed at engineers who deploy LLMs and want help choosing an install method, writing inference code, or starting a trtllm-serve server. After this guide you will know how to install the skill, how to phrase a request that triggers it, and what hardware and accounts you need before running anything it suggests.

TensorRT-LLM Inference ServingOrchestra-Research/AI-Research-SKILLs · MIT Download the skill

Quick start

1Download and unzip the skill. Download the ZIP from this page and unzip it so the tensorrt-llm folder with SKILL.md is in ~/.claude/skills/ or a project's .claude/skills/ folder.
2Or upload it in the Claude apps. In the Claude apps, open Settings, go to Capabilities, then Skills, and upload the ZIP.
3Confirm your hardware. Check that you have an NVIDIA GPU such as an A100 or H100, plus CUDA 13.0.0, TensorRT 10.13.2 and Python 3.10 to 3.12, as the SKILL.md notes list.
4Describe your goal. Ask the agent in plain language, naming your GPU, model and goal, such as serving Llama 3 with batching.
5Review the install commands. Read the Docker or pip commands the agent suggests and confirm they match your environment before running them.
6Start the server or run the code. Run the trtllm-serve command or Python script the agent adapted, then test it with the curl request from the source.

What agent skills are and what this one does

An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches its description. This skill's main file, SKILL.md, gives the agent working knowledge of TensorRT-LLM, NVIDIA's open-source library for optimizing LLM inference.

Once installed, the agent can suggest an install method, write Python code that loads and runs a model, start a trtllm-serve server, and recommend options such as quantization (FP8, INT4), multi-GPU tensor parallelism and batching. It also points you to vLLM or llama.cpp when TensorRT-LLM is not the right fit.

Installation options: Docker image or pip package
Python LLM API for basic inference and batch generation
trtllm-serve for an OpenAI-compatible server
Quantization, parallelism and batching advice, drawn from the reference files

Install the tensorrt-llm skill

Download the skill ZIP from this page and unzip it. For Claude Code, place the unzipped folder in ~/.claude/skills/ to make it available across your projects, or in a project's .claude/skills/ folder to limit it to that project.

In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

Personal install: ~/.claude/skills/
Project install: .claude/skills/ inside your project folder
Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Check your hardware and software before you install TensorRT-LLM

The skill can write install commands, but it cannot give you a GPU. The source lists an NVIDIA GPU (A100, H100 or GB200), CUDA 13.0.0, TensorRT 10.13.2, Python 3.10 to 3.12, and either the tensorrt_llm and torch Python packages or the nvidia/tensorrt_llm Docker image.

Models such as the meta-llama family are downloaded from Hugging Face, which may require an account and acceptance of the model licence.

Not suitable without an NVIDIA GPU, including CPU, Apple Silicon and AMD setups
Large GPU memory is needed for the bigger models in the examples

Trigger the skill with natural requests

You do not need a special command. Describe your goal in plain language that matches the skill's description, such as production LLM deployment on NVIDIA GPUs, quantization, batching or multi-GPU scaling. The agent should then load the skill's guidance.

Be specific about your hardware and model so the agent can pick the right options. Mentioning the GPU type and the model name usually helps most.

"I have 4 H100s. Help me serve Llama 3 8B with TensorRT-LLM."
"Write Python code using TensorRT-LLM to run batch inference on 100 prompts."
"Should I use FP8 quantization for a 70B model on A100s?"
"I'm on a MacBook. Is TensorRT-LLM the right choice, or should I use llama.cpp?"

Start a trtllm-serve server

The serving reference shows trtllm-serve starting an OpenAI-compatible API on a port, and the SKILL.md quick start shows a chat completions request sent with curl. The source notes that the server downloads and compiles the model automatically on first start.

Ask the agent to adapt these flags to your setup. Options in the source include --tp_size for tensor parallelism, --max_batch_size, --max_num_tokens, --dtype and --port. The source also shows an optional --api_key flag for authentication.

Tensor parallelism is set with --tp_size
The source's example request posts to http://localhost:8000/v1/chat/completions

trtllm-serve meta-llama/Meta-Llama-3-8B \
    --tp_size 1 \
    --max_batch_size 256 \
    --port 8000

Use the Python API for quantized and multi-GPU models

For embedding inference in your own code, the skill uses the LLM class from tensorrt_llm with SamplingParams. The source includes examples for FP8 quantization through the dtype argument and for tensor parallelism through tensor_parallel_size.

Treat these examples as starting points. The source's performance figures are not verified, so measure throughput and latency on your own hardware before relying on them.

Basic generation with LLM and SamplingParams
FP8 via dtype="fp8" for H100-class GPUs, as the optimization guide recommends
Multi-GPU via tensor_parallel_size, for example 4 or 8 GPUs

from tensorrt_llm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B",
    dtype="fp8",
    max_num_tokens=8192
)

Safety and what to review before running

This skill runs no scripts, but it does use the network. Model downloads and the Docker or pip installs all fetch files from outside your machine, and the server pulls models automatically on first start. Review every command the agent proposes before running it, especially on shared or production machines.

Treat performance claims with care. The skill's benchmark figures, including the 100x speedup over PyTorch, are unverified and the exact versions it lists may change quickly. Check the current NVIDIA documentation and pin versions yourself.

Read install and download commands before running them
Keep any --api_key value out of shared logs and version control
Confirm the model licence on Hugging Face before downloading

What people use it for

Deciding between Docker and pip installation for an H100 server
Drafting a trtllm-serve command with tensor parallelism and batch size settings
Writing Python code that loads a quantized model and runs batch prompts
Comparing TensorRT-LLM with vLLM or llama.cpp for a specific deployment

Frequently asked questions

Is the tensorrt-llm skill free to use?

The skill is published under the MIT licence, according to its SKILL.md file. The TensorRT-LLM library itself is NVIDIA's open-source project, but you still pay for the NVIDIA GPUs and cloud compute you run it on.

What hardware do I need for TensorRT-LLM?

The source lists NVIDIA GPUs such as A100, H100 or GB200, along with CUDA 13.0.0, TensorRT 10.13.2 and Python 3.10 to 3.12. The skill says TensorRT-LLM does not apply to CPU, Apple Silicon or AMD hardware.

How do I install the tensorrt-llm skill for Claude?

Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, you can upload the ZIP under Settings, then Capabilities, then Skills.

Does the skill run scripts on my machine?

According to the source, the skill runs no scripts, but it uses the network. Model downloads and package installs fetch content from outside your machine, so review commands before running them.

What is the difference between TensorRT-LLM and vLLM?

The SKILL.md suggests vLLM when you want a simpler setup and Python-first API, or PagedAttention without TensorRT compilation, or when you use AMD GPUs. TensorRT-LLM is aimed at maximum throughput and low latency on NVIDIA GPUs.

Why do I need a Hugging Face account for TensorRT-LLM models?

Models such as the meta-llama family are downloaded from Hugging Face, which may require an account and acceptance of the model licence. The server and Python examples in the source load models by their Hugging Face names.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you deploy LLMs on NVIDIA A100 or H100 GPUs and want an agent to draft TensorRT-LLM install steps, Python code and trtllm-serve commands. Skip it if you lack NVIDIA hardware or a GPU budget, or if you need a simpler setup, in which case the skill itself points toward vLLM or llama.cpp.

Download the skill