Skip to content
Get packs

TensorRT-LLM Inference Serving

Guides an agent to set up fast, production LLM serving on NVIDIA GPUs with TensorRT-LLM.

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

This skill gives an AI agent detailed working knowledge of NVIDIA's TensorRT-LLM library, which speeds up running large language models on NVIDIA graphics cards. Once installed, the agent can suggest the right install method, write Python code that loads and runs a model, start a trtllm-serve server, and recommend options such as quantization (FP8/INT4), multi-GPU tensor parallelism, and batching. It also points users to vLLM or llama.cpp when TensorRT-LLM is not the right fit.

Great for

  • Engineers deploying LLMs in production on NVIDIA A100 or H100 GPUs who want higher throughput or lower latency
  • Teams planning multi-GPU or multi-node serving of large models and needing configuration guidance
  • Developers who want an agent to draft TensorRT-LLM serving code and commands

Skip it if

  • Users without NVIDIA GPUs, such as those on CPUs, Apple Silicon or AMD hardware, since the skill itself says TensorRT-LLM does not apply there
  • Beginners with no GPU or cloud budget, because the setup needs CUDA, specific Python versions and large GPU memory
  • Anyone relying on the performance numbers as guaranteed results, since the file's benchmark claims (such as 100x over PyTorch) are unverified marketing-style figures and the exact versions shown may change quickly

Try asking

  • “Set up TensorRT-LLM with Docker and serve Llama 3 8B on 4 GPUs using trtllm-serve”
  • “Write Python code to run batch inference on 100 prompts with TensorRT-LLM”
  • “Show me how to load a 70B model in FP8 with TensorRT-LLM and what memory I will save”
  • “Compare TensorRT-LLM with vLLM for my production workload on H100s”
Step-by-step guide: how to use TensorRT-LLM Inference ServingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    tensorrt-llm.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip tensorrt-llm.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip tensorrt-llm.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Pin the TensorRT-LLM version your environment supports, since the skill references a release candidate (1.2.0rc3) and CUDA/TensorRT versions that may change
  • Check the official docs linked in the skill before running commands on production hardware, and benchmark on your own model rather than relying on the quoted numbers
  • Ask the agent to explain which parallelism and quantization settings fit your GPU memory before launching large models

What's inside

  • references/multi-gpu.md
  • references/optimization.md
  • references/serving.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.