Guide · Agent skill
How to use the TRL LLM Training agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The trl-training skill is a set of instructions that teaches an AI agent how to write and fix code for TRL (Transformers Reinforcement Learning), a Hugging Face library for post-training large language models. It is aimed at people who write training code and want the agent to get trainer classes, dataset formats and config options right. After this guide you will know what the skill changes, how to install it, how to ask for help with a natural request, and what you need on your machine before running anything.
TRL LLM Traininghuggingface/skills · Apache-2.0 Download the skill Quick start
- 1Download the ZIP. Get the trl-training skill ZIP from this page.
- 2Unzip into a skills folder. Extract it so the trl-training folder with SKILL.md sits in ~/.claude/skills/ or your project's .claude/skills/.
- 3Or upload it in the Claude apps. In Settings → Capabilities → Skills, upload the ZIP instead.
- 4Set up TRL in your environment. Install Python with the TRL library and its dependencies, transformers and datasets, on a machine with suitable GPU hardware.
- 5Describe your training goal. Name the method, dataset type and model in a plain request, and include any error text.
- 6Review the generated code. Check the model, dataset source and output directory before running anything.
- 7Run and iterate. Run the script or trl command yourself, then send the traceback back to the agent to fix.
What the trl-training skill makes the agent do
An agent skill is a folder with a SKILL.md file that the agent reads to learn a specific task. This one covers post-training with TRL: supervised fine-tuning (SFT), preference training (DPO and KTO), online reinforcement learning with GRPO, reward-model training, and on-policy distillation.
Once installed, the agent knows which trainer to pick for a given dataset type. SFTTrainer takes language modeling or prompt-completion data, DPOTrainer and KTOTrainer take preference data in different shapes, GRPOTrainer takes prompts plus one or more reward functions, and RewardTrainer trains a scalar reward model rather than a policy. The skill also covers data formatting, memory and speed options, and the trl command-line tool.
Install the trl-training skill
The skill is distributed from the huggingface/skills repository under skills/trl-training. On this page, download the ZIP file for the skill, then unzip it so that the folder containing SKILL.md ends up in one of these locations:
You can also install it for a single project by placing the folder in that project's .claude/skills/ directory. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/trl-training/
- Project install: .claude/skills/trl-training/ inside your project folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Trigger the skill with a natural request
You do not need a special command. Ask for TRL training work in plain language that mentions TRL, the trl CLI, or one of the methods. The skill's description says it applies when you are writing or debugging training code with the TRL Python API or the trl CLI.
Good requests name the goal, the dataset type and the model. Include the error message when debugging, since the agent can then check the dataset format and trainer arguments against the error.
- Write an SFTTrainer script that fine-tunes Qwen/Qwen2.5-0.5B on a chat dataset with assistant-only loss.
- My DPOTrainer run fails with a dataset error. Here is my data format and traceback. Find the problem.
- Which trainer should I use for thumbs-up/thumbs-down feedback with no paired answers? Write the KTO version.
- Set up a GRPO run with a length-based reward function and vLLM in colocate mode.
Example: a basic SFT script
The skill includes a minimal SFT example. It loads a model by ID, passes a training config, loads a dataset from the Hub, and calls train(). Use it as a starting point and ask the agent to adapt the model, dataset and output_dir to your project.
Model loading options go in model_init_kwargs when you pass the model as a string, and LoRA is enabled by passing peft_config.
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B", # model ID or a PreTrainedModel instance
args=SFTConfig(output_dir="Qwen2.5-0.5B-SFT"),
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()Get the dataset format right
Most training failures come from data shape. The skill describes two conversational formats: a messages list with role and content fields for language modeling, and prompt and completion lists for prompt-completion training. The chat template is applied automatically, so the agent is told not to apply it to your data yourself. Extra columns are allowed, and GRPO passes them through to your reward functions.
Tune SFT and GRPO options
For SFT, the skill highlights max_length for truncation, packing for fewer padding tokens, padding_free, use_liger_kernel for lower peak memory, and assistant_only_loss for training only on assistant turns. Packing and padding_free have hardware requirements noted in the source, so check them before enabling.
For GRPO, the generation batch must be divisible by num_generations (default 8). Generation is usually the bottleneck, so the skill recommends vLLM. Use vllm_mode="colocate" to share the training GPUs, or vllm_mode="server" with a separate trl vllm-serve process.
- Reward functions receive prompts, completions, completion_ids and trainer_state, plus any extra dataset columns; accept **kwargs for ones you ignore.
- Return one float per completion as a list.
- With conversational data, completions is a list of message lists rather than strings.
def reward_len(completions, **kwargs):
return [-abs(20 - len(c[0]["content"])) for c in completions]Use the trl command line
The trl CLI mirrors the config fields as flags. For example, you can run an SFT job with a model name and a dataset name. Configs can be loaded from YAML with --config, and distributed presets are available with --accelerate_config. Python scripts are launched with accelerate launch.
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/CapybaraWhat people use it for
- Writing an SFT script for a chat dataset with assistant-only loss.
- Debugging a DPO run that fails because the preference data has the wrong columns.
- Choosing between DPO and KTO for a project with unpaired thumbs-up or thumbs-down labels.
- Setting up a GRPO run with a custom reward function and vLLM generation.
- Training a reward model with RewardTrainer on chosen and rejected pairs.
Frequently asked questions
Is the trl-training skill free?
The skill is licensed under Apache-2.0, as stated in its SKILL.md metadata. The TRL library is a separate Hugging Face project, and this page does not state pricing for anything beyond the skill itself.
What do I need to use the trl-training skill?
You need Python with the TRL library and its dependencies, transformers and datasets. For practical training runs the skill calls for GPU hardware. vLLM is optional for faster GRPO generation, and FlashAttention or Liger kernels are optional for some SFT options.
Does the skill run training for me?
No. The skill guides the code the agent writes, and it does not run training from this file alone. You run the script or trl command yourself.
How do I install the trl-training skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, you can upload the ZIP under Settings → Capabilities → Skills.
Should I use DPO, KTO or GRPO?
Use DPO for paired chosen and rejected answers, and KTO when you only have a per-sample good or bad label. GRPO fits online reinforcement learning when you have prompts and a reward function. Describe your data to the agent and it can recommend a trainer.
Why does my TRL trainer fail with a dataset error?
Dataset shape is the most common cause. Check that your data matches the format for your trainer, such as messages or prompt and completion lists for conversational data. Paste the full traceback into your request so the agent can compare it with the trainer's expected columns.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits machine learning engineers and developers who write TRL training code and want help choosing trainers, fixing dataset formats and setting memory options. It adds little for people who only chat with a model, and the experimental trainers in trl.experimental may change without notice, so pick stable trainers for production work.
Download the skill