Skip to content
Get packs

Guide · Agent skill

How to use the SimPO Preference Training agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The simpo-training skill teaches an AI agent how to fine-tune a large language model with SimPO (Simple Preference Optimization), a method that aligns a model using pairs of preferred and rejected answers. This guide explains what the skill does, how to install it, how to ask for help with natural requests, and what hardware, data and network access you need. By the end you will know whether it fits a preference-training project you already have.

SimPO Preference TrainingOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Get the skill files. Download the skill ZIP from this directory page. Keep the SKILL.md file and the references folder together.
  2. 2Install the skill. Unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Prepare the environment. Create a Python 3.10 conda environment, install PyTorch 2.2.2, clone and pip install alignment-handbook, then install Flash Attention 2.
  4. 4Prepare preference data. Format your data with prompt, chosen and rejected fields, as described in the skill's dataset reference.
  5. 5Ask the agent for a config. Describe your model, data and goal in plain language, and ask for a SimPO config with suggested beta and learning rate.
  6. 6Launch and review. Run the accelerate launch command the agent gives you, then check loss and memory, using the troubleshooting notes if the run diverges or runs out of memory.

What does the simpo-training skill make the agent do?

Agent skills are folders of instructions that an AI agent loads when a task matches them. This one covers preference alignment with SimPO. Its description says SimPO is a reference-free alternative to DPO that needs no separate reference model, which the skill presents as simpler and more efficient.

Once installed, the agent can write training configs, choose hyperparameters such as beta and learning rate, suggest datasets, and walk through troubleshooting for problems such as loss divergence or out-of-memory errors.

  • Write YAML training configs for SimPO runs
  • Explain and pick values for beta, gamma_beta_ratio, learning rate and sft_weight
  • Suggest preference datasets and explain the chosen/rejected format
  • Troubleshoot loss divergence, forgotten capabilities, weak preference separation and OOM errors

How do you install the simpo-training skill?

Download the skill ZIP from this directory page and unzip it into one of two locations. Use ~/.claude/skills/ to make it available across your projects, or a project's .claude/skills/ folder to limit it to that project.

If you use the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into the project's .claude/skills/ folder
  • Claude apps: Settings → Capabilities → Skills, then upload

What do you need before you can train with SimPO?

The skill is a guide for training, not a compute provider. The source lists Python 3.10 with conda, PyTorch 2.2.2, the alignment-handbook library, Flash Attention 2, Hugging Face libraries, DeepSpeed with a ZeRO-3 config, and NVIDIA GPUs (A100 or H100 recommended). Training 7B+ models needs large GPUs; the skill's hardware notes list 1× A100 40GB for a 7B model with DeepSpeed ZeRO-3 and 8× A100 80GB for 70B.

You also need chosen/rejected preference data and access to any gated models you use, such as Meta Llama 3 or Mistral weights on Hugging Face, which may require an account and token.

  • Python 3.10 and conda
  • PyTorch 2.2.2 and Flash Attention 2
  • alignment-handbook, cloned from GitHub and pip installed
  • transformers, datasets, trl, accelerate and DeepSpeed
  • NVIDIA GPU(s), A100/H100 recommended
  • Preference dataset with prompt, chosen and rejected fields
  • Hugging Face token for gated models, if needed
conda create -n simpo python=3.10 && conda activate simpo git clone https://github.com/huggingface/alignment-handbook.git cd alignment-handbook python -m pip install . python -m pip install flash-attn --no-build-isolation

How do you trigger the skill with natural requests?

You do not need special syntax. Describe the alignment job in plain language and mention preferences, chosen/rejected pairs or SimPO. The agent should load the skill when the request matches its description.

Good requests name the model, the data and the problem. The skill's own workflows cover a base model (Mistral 7B), an instruct model (Llama 3 8B) and a math or code task with a lower learning rate.

  • Write a SimPO config to fine-tune Mistral-7B-v0.1 on UltraFeedback preference pairs.
  • My SimPO run shows loss divergence. What should I change first?
  • Suggest beta and learning rate for an 8B instruct model with a small preference set.
  • Compare SimPO and DPO setups for a team with limited compute.

What does a SimPO training run look like?

The skill's base-model workflow uses a YAML config with the model name, a dataset mixer, SimPO hyperparameters and output directory, then launches a DeepSpeed ZeRO-3 run through accelerate. Its example uses beta 2.0, gamma_beta_ratio 0.5, loss_type sigmoid and learning_rate 5e-7 as starting points, and the skill notes the learning rate is critical in the 3e-7 to 1e-6 range.

Check the values against your own data before running. The skill's configs are presented as examples and are not tested by this file.

  • Base model example: mistralai/Mistral-7B-v0.1 with HuggingFaceH4/ultrafeedback_binarized
  • Instruct model example: meta-llama/Meta-Llama-3-8B-Instruct with a Llama 3 8B config and sft_weight 0.1
  • Reasoning example: deepseek-ai/deepseek-math-7b-base with beta 5.0 and learning rate 3e-7
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml

What are the safety notes for scripts and network access?

This skill runs scripts and uses the network. The launch command runs scripts/run_simpo.py, the install steps clone code from GitHub and pip install packages, and training downloads models and datasets from Hugging Face. Review the scripts and commands before you run them, especially on a shared machine or cluster.

Gated models may require a Hugging Face account and token. Keep tokens out of shared config files and output folders.

  • Runs scripts: scripts/run_simpo.py and accelerate launch commands
  • Uses network: git clone from GitHub, pip installs, Hugging Face downloads
  • Review install and launch commands before running them
  • Do not place access tokens in configs that you share

What people use it for

  • Aligning an open 7B or 8B model with an existing chosen/rejected dataset
  • Comparing a SimPO configuration against a DPO or PPO setup for the same data
  • Getting a starting config and hyperparameter explanations for a single-node training run
  • Diagnosing a run with loss divergence or out-of-memory errors

Frequently asked questions

How do I install the simpo-training skill?

Download the skill ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills instead.

What hardware do I need to use SimPO training?

The skill recommends NVIDIA A100 or H100 GPUs. Its notes list 1× A100 40GB for a 7B model with DeepSpeed ZeRO-3 and 8× A100 80GB for a 70B model.

Is the simpo-training skill free and what is its licence?

The skill is listed under the MIT licence. The compute you need for training is not included and has to come from your own hardware or provider.

What is the difference between SimPO and DPO?

According to the skill, SimPO is reference-free, so it does not need a separate reference model, while DPO does. The skill describes SimPO as simpler and more efficient, and its performance claim of +6.4 points on AlpacaEval 2.0 comes from the method's paper.

Can I use the skill instead of PPO or GRPO?

The skill suggests SimPO when you want simpler training than DPO and have preference pairs. It describes PPO as giving maximum control but needing a reward model and a more complex setup, and GRPO as memory-efficient RL with no critic.

What do I do if SimPO training diverges or runs out of memory?

For loss divergence, the skill suggests lowering the learning rate (for example from 5e-7 to 3e-7) or reducing beta. For out-of-memory errors, it suggests a smaller per-device batch size with more gradient accumulation and enabling gradient checkpointing.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits ML engineers who already have chosen/rejected data and want a clear SimPO recipe for an open model on a single node. It is not a good choice if you lack GPU access or preference data, or if you need a verified, turnkey result, since the configs are not tested by the file and outcomes depend on your data and setup.

Download the skill