Skip to content
Get packs

Guide · Agent skill

How to use the TRL Fine-Tuning Guide agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The fine-tuning-with-trl skill gives an AI agent a detailed playbook for post-training language models with the HuggingFace TRL library. Once installed, the agent can draft and run code for supervised fine-tuning, DPO preference alignment, reward model training and online RL with PPO or GRPO. This guide is for readers who have never used agent skills and want to install this one, trigger it and understand what it can and cannot do.

TRL Fine-Tuning GuideOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Get the fine-tuning-with-trl skill ZIP from the page where you found this guide.
  2. 2Unzip into a skills folder. Extract the archive into ~/.claude/skills/ for personal use, or into .claude/skills/ in a project folder.
  3. 3Install the Python libraries. Run the pip command for trl, transformers, datasets, peft and accelerate in the environment where you will train. Torch is also required.
  4. 4Describe your training goal. Ask the agent in plain language, for example for DPO on a preference dataset, and name your model and data.
  5. 5Review the generated code. Check the model names, dataset names and training arguments before running anything.
  6. 6Run a small test first. Use a small model and few steps to confirm the setup works on your GPU.

What does the skill make the agent do?

The skill is a set of written instructions and reference notes that the agent reads when your request matches its description. It does not contain a trained model. It teaches the agent the TRL workflows and dataset formats so it can write training code for you.

The source covers four areas: supervised fine-tuning (SFT) for instruction tuning, DPO for preference alignment, reward model training, and online reinforcement learning with PPO or GRPO. It also includes a full RLHF pipeline that chains SFT, a reward model, and PPO.

  • SFT: train a base model on instruction or prompt-completion data.
  • DPO: align a model using chosen/rejected preference pairs, without a separate reward model.
  • Reward modeling: train a model to score completions based on human preferences.
  • PPO and GRPO: optimize a policy against a reward signal during training.

How do you install the skill?

Download the skill ZIP from the page where you found this guide. Then choose one of the two install options below.

For a personal install, unzip the file into ~/.claude/skills/. For a single project, unzip it into the .claude/skills/ folder inside that project. You can also upload the skill in the Claude apps under Settings → Capabilities → Skills.

  • Personal: unzip into ~/.claude/skills/
  • Project only: unzip into .claude/skills/ in your project folder
  • Claude apps: Settings → Capabilities → Skills, then upload the skill

What do you need before the skill is useful?

The skill only writes and guides code. The training runs on your own machine, so you need the Python libraries it depends on. The source lists trl, transformers, datasets, peft, accelerate and torch, and gives this install command.

Practical training runs need GPU hardware. Downloading base models and datasets, such as the trl-lib datasets, requires HuggingFace Hub access, so the skill uses the network.

  • Python with pip
  • Libraries: trl, transformers, datasets, peft, accelerate, torch
  • GPU hardware for practical training runs
  • HuggingFace Hub access to download models and datasets
pip install trl transformers datasets peft accelerate

How do you trigger the skill with a natural request?

You do not need a special command. Describe the training goal in plain language and mention what you are trying to achieve. The skill's description lists alignment with preferences, RLHF, and training from human feedback as triggers, so use those ideas in your request.

Name the method if you know it (SFT, DPO, GRPO, PPO, reward model). If you do not, describe the goal and let the agent choose the workflow.

  • Write a DPO training script for my preference dataset, using beta 0.1.
  • Help me set up a reward model on the trl-lib/ultrafeedback_binarized dataset.
  • I want to align a small Qwen model with human preferences. What pipeline should I use?
  • Draft a GRPO setup with a reward function that favors concise answers.

What do the outputs look like?

For DPO, the source expects a preference dataset where each row has a prompt, a chosen answer and a rejected answer. The skill shows the agent how to load such a dataset from the trl-lib hub or from your own JSON file.

The CLI alternative for DPO and GRPO is also shown in the source. Ask the agent to produce either the Python version or the trl command, depending on how you prefer to run jobs.

trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct --dataset_name argilla/Capybara-Preferences --output_dir Qwen2.5-0.5B-DPO --per_device_train_batch_size 4 --learning_rate 5e-7 --beta 0.1

What safety and cost notes should you know?

The skill runs scripts and uses the network. Read any code the agent writes before running it, and check the model and dataset names it points to. Training jobs can use significant GPU time and memory, so start with a small model and a short run.

The evaluation step in the skill is a single example generation. It does not check safety or output quality rigorously, so do your own evaluation before relying on a model.

  • Review generated scripts before running them.
  • Check which models and datasets are downloaded from the Hub.
  • Start with small models and few steps to limit compute use.
  • Do not treat the single-prompt test as a real evaluation.

Tips for better results

Give the agent your dataset format and hardware limits in the first message. The skill's workflows depend on the right fields, such as prompt, chosen and rejected for DPO, so stating them up front avoids rework.

If you want to learn the differences between pipelines, ask the agent to explain each step before writing code. The reference notes cover DPO loss variants, online RL settings and reward modeling in more detail.

  • Name your dataset and its columns.
  • State your GPU memory and batch size limits.
  • Ask for one method at a time when learning.
  • Ask for explanations of key parameters such as beta, kl_coef and num_generations.

What people use it for

  • Drafting a DPO script to align a small instruct model with chosen/rejected pairs.
  • Setting up a reward model training run on a preference dataset before PPO.
  • Writing a GRPO reward function that scores completions for length or other criteria.
  • Learning how SFT, DPO and RLHF pipelines differ by asking the agent to walk through each stage.

Frequently asked questions

Is the TRL Fine-Tuning Guide skill free?

The skill is published under the MIT licence, as stated in its source. The HuggingFace libraries it references are used through your own environment, and you pay for any GPU or compute you use to run training.

How do I install the fine-tuning-with-trl skill?

Download the ZIP from the page, then unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ in a project. You can also upload it in the Claude apps under Settings → Capabilities → Skills.

What does the skill require?

It needs Python with pip and the trl, transformers, datasets, peft, accelerate and torch libraries. Practical training needs GPU hardware and HuggingFace Hub access to download models and datasets.

Does the skill train a model for me?

No. The skill instructs the agent to write and guide training code. The training itself runs on your machine using the libraries you installed.

What is the difference between SFT, DPO and RLHF?

SFT trains a model on instruction or prompt-completion examples. DPO aligns a model using chosen and rejected answer pairs without a separate reward model. RLHF in the source means a pipeline that adds a reward model and reinforcement learning such as PPO after SFT.

Can I use this skill for production alignment work?

The source says its evaluation step is a single example generation and does not check safety or quality rigorously. Pair it with your own evaluation before using a model in production.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are an ML engineer or researcher who wants an agent to draft TRL training scripts and learn the SFT, DPO and RL workflows. Pick something else if you lack GPU access or ML experience, or if you need a managed service that fine-tunes models for you and checks their safety.

Download the skill