Guide · Agent skill
How to use the Preference Optimization agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The preference-optimization skill is a decision guide for an AI agent that is aligning a fine-tuned language model with preference data. It helps the agent choose among DPO, ORPO, KTO and SimPO based on the shape of your feedback, suggests starting hyperparameters, describes an iterative on-policy training pattern and explains how to build chosen/rejected pairs from graded attempts. This guide walks through what the skill does, how to install it, how to trigger it and what to check before you rely on it.
Preference Optimizationwshobson/agents · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or the project's .claude/skills/ folder for one project.
- 2Check your prerequisites. Confirm you have a fine-tuned SFT checkpoint, preference data or thumbs-up/down feedback, and a Python ML training stack such as TRL and Unsloth.
- 3Describe your data in plain language. Tell the agent what feedback you have, such as paired preferences or unpaired thumbs-up/down labels, and ask which method fits.
- 4Accept the starting hyperparameters. Use the suggested DPO beta and learning rate as a starting point, and check that the learning rate is lower than your SFT learning rate.
- 5Build pairs from same-task attempts. Pair passing and failing attempts at the same task, choosing the rejected member at mu minus 2 sigma of the reward distribution.
- 6Plan iterative rounds and validate. Plan more than one on-policy round and re-check the chosen method at your deployment model size before relying on it.
What the skill makes the agent do
The skill is mostly guidance. Its core is a method selection table: preference pairs go to DPO by default, memory-bound cases or missing SFT checkpoints go to ORPO, unpaired thumbs-up/down feedback goes to KTO, and length bias with sweep budget goes to SimPO.
It also tells the agent how to pick starting values, such as a DPO beta of 0.1 and a learning rate of 5e-7 to 1e-6 for 1 to 2 epochs, and warns that this learning rate is lower than the SFT learning rate that produced the checkpoint.
- Choose a method from the shape of your preference data
- Suggest starting hyperparameters
- Describe an iterative, on-policy DPO loop
- Explain how to build chosen/rejected pairs from graded trajectories
Install the skill from the ZIP
Download the ZIP for this skill from this page and unzip it. The skill folder contains SKILL.md and references/method-configs.md.
For a personal install, unzip it into ~/.claude/skills/. For a single project, unzip it into the project's .claude/skills/ folder. Alternatively, upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/
- Project: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Trigger the skill with natural requests
You do not need a special command. Describe your situation in plain language and mention preference data, thumbs-up/down feedback, method choice or a DPO run problem. The skill's description names these triggers: preference pairs or thumbs-up/down feedback exist, you are choosing between methods, or a DPO run needs hyperparameters or debugging.
- "We have an SFT checkpoint and clean paired preference data. Which method should we use?"
- "Reviewers click thumbs-up or thumbs-down on each response and nothing is paired. Should we use DPO?"
- "My DPO run's loss looks off. Are the learning rate and beta reasonable?"
- "Our DPO model favors longer answers regardless of quality. What should we try?"
Work through a typical example
Suppose reviewers rate each response with a thumbs-up or thumbs-down and nothing is paired. The skill routes this to KTO, because KTO takes a binary label per example. The skill explicitly says not to synthesize pairs just to force DPO onto unpaired data.
If you have clean paired data and an SFT checkpoint, the default is DPO at beta 0.1. If your GPU budget does not cover a separate SFT pass plus a DPO reference model, the skill points to ORPO, which is reference-free and combines the SFT and preference objectives in one loss.
Build chosen and rejected pairs
Build DPO or ORPO pairs from two attempts at the same task, one passing and one failing. Do not pair unrelated best and worst examples from different tasks. Choose the rejected member at mu minus 2 sigma of the reward distribution, not the absolute minimum, because naive best-versus-worst pairing degrades as scale increases.
For the mechanics of turning graded traces into pairs, including rejection sampling and judge-scored selection, use the trace-to-training-data skill.
sorted_by_reward = sort(trajectories, key=reward)
chosen = sorted_by_reward[-1] # highest reward
mu, sigma = mean(rewards), stdev(rewards)
rejected = closest(sorted_by_reward, mu - 2 * sigma)Use the iterative on-policy pattern
A single offline DPO pass is a starting point, not the full production pattern. The skill describes a loop: sample completions from the current checkpoint, score or rank them with a reward model, judge or task grader, run a DPO pass using the current checkpoint as the reference, then make the result the new policy and new reference for the next round.
The skill advises planning for at least one more round after the first checkpoint exists.
- Sample completions from the current policy checkpoint
- Score or rank the completions
- Run a DPO pass with the current checkpoint as reference
- Repeat with the new checkpoint as both policy and reference
Check the configs and debug a DPO run
The file references/method-configs.md holds complete TRL config blocks for DPOConfig, ORPOConfig and KTOConfig, plus the SimPO sweep grid and a catastrophic-forgetting note. The config examples use BASE_MODEL and SFT_CHECKPOINT placeholders, so you must choose the real checkpoint yourself. The skill uses processing_class rather than tokenizer= in trainer calls.
If a preference-tuned model loses general capability, the skill says a too-high learning rate is the usual cause. Its advice is to lower the learning rate toward the low end of the recommended range before trying other fixes.
- DPO: beta 0.1, learning rate 7e-7 in the example config, 2 epochs
- Use bf16 rather than fp16, as the config notes
- Lower the learning rate first if forgetting appears
What people use it for
- Choosing between DPO and KTO for a team with thumbs-up/down feedback
- Picking ORPO when GPU memory cannot hold a separate reference model
- Debugging a DPO run whose learning rate or beta looks wrong
- Building chosen/rejected pairs from graded agent trajectories
- Planning a second on-policy DPO round after a first checkpoint
Frequently asked questions
How do I install the preference-optimization skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
What do I need before using this skill?
You need a fine-tuned SFT checkpoint and preference data, either paired preferences or thumbs-up/down feedback. The skill references a Python ML training stack such as TRL and Unsloth, which the files do not verify on their own.
Is this skill free?
The skill is published under the MIT licence, as listed on this page. The source does not mention pricing, so check the repository for any terms beyond the licence.
DPO or KTO: which should I use?
Use DPO when you have matched preference pairs, since that is the default. Use KTO when feedback is unpaired thumbs-up/down, and do not force unpaired data into synthetic pairs.
Does the skill run scripts or use the network?
According to this listing, the skill does not run scripts and does not use the network. It is mainly guidance plus a config reference file.
Why does my DPO model lose general ability?
The skill says a too-high learning rate is the usual cause. Lower the learning rate toward the low end of the recommended range before trying other remedies.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits ML engineers who already have an SFT checkpoint and preference or thumbs-up/down data, and teams deciding between DPO and simpler unpaired options like KTO. It is not for people without preference data or a fine-tuned model, and it is not a verified ranking of methods, so validate any variant choice at your target model size before committing.
Download the skill