Skip to content
Get packs

Guide · Agent skill

How to use the OpenRLHF training agent skill (openrlhf-training)

Updated · written with AI from the project's own documentation and checked against our catalogue data

The OpenRLHF training skill (skill name openrlhf-training) gives an AI agent working knowledge of OpenRLHF, an open reinforcement learning from human feedback framework for large language models built on Ray and vLLM. It is aimed at ML engineers who have multi-GPU infrastructure and want help planning, writing or adapting setup and launch commands. After this guide you will know what the skill does, how to install it, how to ask for it in natural language, and what hardware and safety checks to handle before running anything.

OpenRLHF Reinforcement TrainingOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download the ZIP. Download the openrlhf-training skill ZIP from this page.
  2. 2Install the skill. Unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Prepare your hardware. Confirm you have NVIDIA GPUs with CUDA, Docker with the nvidia runtime, and a Ray cluster for multi-GPU jobs.
  4. 4Make a natural request. Describe your goal, model size, GPU count and algorithm, such as PPO, GRPO, RLOO or DPO.
  5. 5Review the generated commands. Check model names, dataset paths, output directories and GPU allocation before running anything.
  6. 6Start small and monitor. Run a short job first to check memory use and throughput before a full training run.

What is an agent skill, and what does this one do?

An agent skill is a folder of instructions and reference files that an AI agent loads when a task matches the skill's description. This one is built around OpenRLHF, which uses Ray for distributed training and vLLM for fast generation during RLHF.

Once installed, the agent can plan and write the setup and launch commands for OpenRLHF. It knows the typical steps: installing the Docker image and packages, starting a Ray cluster, choosing an algorithm (PPO, GRPO, RLOO or DPO), and tuning GPU allocation, batch sizes and learning rates. It also knows common fixes for out-of-memory errors and training instability.

  • Algorithms covered: PPO, GRPO, RLOO and DPO, plus variants listed in the reference notes such as REINFORCE++ and Dr. GRPO.
  • Reference files cover the Hybrid Engine (GPU sharing), algorithm comparison, multi-node Ray setup, and custom reward functions.
  • Target scale in the source: large models from 7B to 70B+.

How do you install the openrlhf-training skill?

Download the ZIP for the skill from this page. Then install it using one of the two methods below.

For Claude Code, unzip the folder into ~/.claude/skills/ for all your projects, or into a project's .claude/skills/ folder to limit it to that project. For the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into .claude/skills/ inside the project folder
  • Claude apps: Settings → Capabilities → Skills → upload

What do you need before using it?

The skill assumes you already have the infrastructure. Check these before you ask the agent to launch anything.

The source lists NVIDIA GPUs with CUDA and Docker with the nvidia runtime, Python packages openrlhf (with the vLLM extra), ray, vllm, torch, transformers and deepspeed, and a Ray cluster for multi-GPU jobs. Some models, such as meta-llama/Meta-Llama-3-8B, may require a Hugging Face account and license acceptance.

The source's hardware notes recommend NVIDIA A100 or H100 GPUs. It gives 8× A100 40GB for a 7B model with the Hybrid Engine, and 48× A100 80GB for a 70B model with a vLLM:Actor:Critic ratio of 1:1:1. Multi-node setups are recommended to use InfiniBand.

  • NVIDIA GPUs with CUDA and Docker (nvidia runtime)
  • Python packages: openrlhf[vllm], ray, vllm, torch, transformers, deepspeed
  • Ray cluster for multi-GPU jobs
  • Hugging Face access for gated models

How do you trigger the skill with natural requests?

You do not need special syntax. Describe the job in plain terms that mention RLHF, the algorithm, or OpenRLHF, and include your model size and GPU count. The agent will match the request to the skill.

Good requests state the goal, the hardware and the algorithm. Vague requests produce generic advice, so give the agent your real numbers.

  • "Write an OpenRLHF PPO command for Llama 3 8B on one node with 8 A100s."
  • "Set up GRPO with OpenRLHF without a critic model, and explain the memory tradeoffs."
  • "My OpenRLHF PPO run runs out of memory on 8 GPUs. What should I change?"
  • "Compare DPO and PPO in OpenRLHF for my preference dataset."

What does a typical PPO or GRPO command look like?

The skill draws on commands from its reference. A PPO run starts a Ray head node, then submits a training job with the train_ppo_ray module. GRPO uses the same command with the advantage estimator set to group_norm. The example below is the source's PPO quick start; check every value against your own hardware before running it.

  • GRPO: add --advantage_estimator group_norm to the PPO command.
  • DPO: uses deepspeed --module openrlhf.cli.train_dpo, with no reward model.
ray start --head --node-ip-address 0.0.0.0 --num-gpus 8 ray job submit --address="http://127.0.0.1:8265" \ --runtime-env-json='{"working_dir": "/openrlhf"}' \ -- python3 -m openrlhf.cli.train_ppo_ray \ --ref_num_nodes 1 --ref_num_gpus_per_node 8 \ --reward_num_nodes 1 --reward_num_gpus_per_node 8 \ --critic_num_nodes 1 --critic_num_gpus_per_node 8 \ --actor_num_nodes 1 --actor_num_gpus_per_node 8 \ --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \ --colocate_all_models \ --pretrain OpenRLHF/Llama-3-8b-sft-mixture \ --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \ --save_path ./output/llama3-8b-rlhf

Which algorithm should you pick?

The skill's notes give a short rule of thumb. PPO gives the most control and suits complex rewards, but needs a critic and uses more memory. GRPO is more memory-efficient and needs no critic. RLOO is a modified PPO with per-token KL. DPO is the simplest because it needs no reward model. The notes also name TRL, veRL and DeepSpeedChat as alternatives for different situations.

  • PPO: maximum control, best for complex rewards
  • GRPO: memory-efficient, no critic needed
  • RLOO: modified PPO with per-token KL
  • DPO: simplest, no reward model needed
  • TRL: single-node training with a simpler API

How do you fix common problems?

The skill lists fixes for the errors people hit most often. Ask the agent to walk through the relevant one with your exact error message.

For GPU out-of-memory errors, the source suggests removing --colocate_all_models and allocating separate GPUs to each model. For a DeepSpeed GPU index error, it suggests setting RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1. For instability, it suggests using the Hybrid Engine flags and raising --init_kl_coef from 0.01 toward 0.05. For slow PPO generation, it suggests enabling vLLM engines.

  • Out of memory: disable colocation and give each model its own GPUs
  • DeepSpeed GPU index out of range: set RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1
  • Instability: use the Hybrid Engine flags and adjust the KL coefficient
  • Slow generation: enable vLLM engines

What safety checks should you run first?

This skill does not run scripts itself, but it uses the network, and the commands it writes launch long, costly GPU jobs. Review every command before you run it. Check the model names, dataset names, output paths and GPU counts against your own cluster.

The source makes performance claims, including that OpenRLHF is 2× faster than DeepSpeedChat. The skill does not verify these, so treat them as the project's claims rather than your results. Benchmark on your own workload.

  • Read each command before running it; a misconfigured job can waste expensive compute.
  • Confirm model licenses and Hugging Face access before launching.
  • Start with a small run to check memory and throughput.
  • Do not treat the skill's speed claims as verified on your hardware.

What people use it for

  • Writing a starting PPO command for a 7B model on a single 8-GPU node.
  • Setting up a GRPO run without a critic model to reduce memory use.
  • Adapting a DPO command to a different preference dataset with --dataset and --chosen_key options.
  • Diagnosing an out-of-memory error by reviewing colocation and GPU allocation settings.
  • Planning a multi-node Ray cluster using the multi-node reference notes.

Frequently asked questions

Is the OpenRLHF training skill free to use?

The skill is published under the MIT license, as listed on this page. OpenRLHF itself is an open framework, but you still pay for the GPU infrastructure your training runs use.

What hardware do I need for the openrlhf-training skill?

The source recommends NVIDIA A100 or H100 GPUs, with CUDA and Docker using the nvidia runtime. Its examples use 8 GPUs per node for a 7B model and 48 A100 80GB GPUs for a 70B model.

Can I run OpenRLHF on a single consumer GPU?

The skill's own notes are not suited to small experiments on a single consumer GPU. They suggest TRL for that case, since OpenRLHF training would be slow or fail on memory.

How do I install the skill in Claude?

Download the ZIP and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, upload it under Settings → Capabilities → Skills.

What is the difference between PPO, GRPO and DPO in OpenRLHF?

PPO gives the most control but needs a critic and more memory. GRPO is memory-efficient and needs no critic, and DPO is the simplest because it needs no reward model.

Why does my OpenRLHF job run out of GPU memory?

The skill suggests removing --colocate_all_models and allocating separate GPUs to the actor, critic, reward and reference models. Ask the agent to review your exact GPU counts and error message.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits ML engineers with multi-GPU clusters who want to fine-tune 7B to 70B+ models with PPO, GRPO, RLOO or DPO and need a starting configuration. It is a poor fit for single-GPU experiments, teams without GPU infrastructure or RL experience, or production training that has not been reviewed, because the commands launch costly jobs and the speed claims are not verified.

Download the skill