Skip to content
Get packs

Preference Optimization

Choose and configure DPO, ORPO, KTO or SimPO to align a fine-tuned model with preference feedback.

Source Download skill
Repository
wshobson/agents
Stars
42,095
Licence
MIT
Files
2 files

What it does

It gives an AI agent a decision guide for aligning a fine-tuned language model with human-style preference data. The agent picks among methods (DPO, ORPO, KTO, SimPO) based on the data shape, suggests starting hyperparameters, explains an iterative on-policy training pattern, and describes how to build chosen/rejected training pairs from graded attempts. The skill itself is mostly guidance; the referenced config file holds TRL config blocks.

Great for

  • ML engineers who already have an SFT checkpoint and thumbs-up/down or pairwise preference data
  • Teams deciding between DPO and simpler or unpaired alternatives like KTO
  • Anyone debugging a DPO run whose learning rate or beta looks off

Skip it if

  • People without preference data or a fine-tuned model: the skill assumes both exist and that routing to this method has already happened, so it will not help with basic fine-tuning.
  • Teams looking for a verified 'best method' ranking: the skill itself cites evidence that variant rankings can invert with model scale, so variant advice should be validated at the target size.
  • Users wanting to run training unsupervised on production models: the workflow involves sampling, scoring and repeated training rounds that need compute and careful evaluation.

Try asking

  • “We have an SFT checkpoint and clean paired preference data. Which method should I use and at what beta?”
  • “Our reviewers only give thumbs-up/down per response, not pairs. Should we use DPO?”
  • “Our DPO model prefers longer answers regardless of quality. Help me choose between SimPO and other fixes.”
  • “Help me build chosen/rejected pairs from graded agent trajectories for DPO.”
Step-by-step guide: how to use Preference OptimizationInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    preference-optimization.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip preference-optimization.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip preference-optimization.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Start with DPO at beta 0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs; do not reuse the SFT learning rate.
  • Re-validate any method ranking at the actual deployment model size before relying on it.
  • Plan more than one on-policy round after the first DPO checkpoint rather than treating one pass as final.

What's inside

  • references/method-configs.md
  • SKILL.md

Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.