How to use the run-experiment agent skill for GPU servers

What the run-experiment skill makes the agent do

An agent skill is a folder containing a SKILL.md file with instructions the agent reads when a request matches the skill's description. The run-experiment skill's description says it deploys and runs ML experiments on local or remote GPU servers, and it triggers on phrases such as "run experiment", "deploy to server", or a request to launch training jobs.

Once triggered, the skill works as a checklist rather than a blind launch. The agent reads your project's AGENTS.md to learn the environment, checks GPU memory on the target machine, syncs code, optionally adds Weights & Biases logging, starts the job, and reports back which GPU and screen session it used.

• Detects the environment from AGENTS.md: local GPU, remote SSH server, Vast.ai instance, or Modal serverless
• Treats a GPU as free when memory used is under 500 MiB
• Syncs only code files by default, not data or checkpoints
• Runs each experiment in its own screen session on its own GPU
• Uses tee to save logs for later inspection
• Optionally sends a Feishu notification and destroys a Vast.ai instance after results are saved

Install the run-experiment skill

Download the ZIP for this skill from our page. The skill's files are in a folder named run-experiment that contains SKILL.md. Install it in one of two ways.

For Claude Code, unzip the folder into ~/.claude/skills/ to make it available for all your projects, or into a project's .claude/skills/ folder to limit it to that project.

For the Claude apps, upload the skill under Settings → Capabilities → Skills.

• Personal install: unzip into ~/.claude/skills/
• Project install: unzip into the project's .claude/skills/
• Claude apps: Settings → Capabilities → Skills, then upload

Set up your project's AGENTS.md

The skill finds your compute environment by reading AGENTS.md in your project. If it finds no server information, the agent is instructed to ask you. Add a section that describes where experiments run. The skill's own example uses a Remote Server heading with the SSH alias, GPU details, conda activation command, code directory, and code_sync setting.

Other settings the skill looks for include wandb (true or false), and for Vast.ai, gpu: vast, vast_instance, SSH host and port, remote path, and auto_destroy. For Modal, look for gpu: modal and the app or function name.

• SSH alias, conda activation command, and code directory for a lab server
• code_sync: rsync (default) or git
• wandb: true to add Weights & Biases logging
• gpu: vast with vast_instance and auto_destroy for Vast.ai
• gpu: modal with app/function name for Modal

Trigger the skill with natural requests

You do not need special syntax. Describe the experiment in plain language and include one of the trigger phrases. The skill accepts the arguments you type after its name as the experiment description.

Be specific about the script, the machine, and any arguments, so the agent does not have to ask. If AGENTS.md is missing details, the agent will ask you for them rather than guess.

• "Run experiment: train the baseline in train.py on my lab server"
• "Deploy to server the eval sweep with 3 seeds, one per free GPU"
• "Launch the training job on the Vast.ai instance and destroy it when results are saved"
• "跑实验: fine-tune with config configs/base.yaml"

What the agent does during a run

For a remote server, the agent first checks GPUs with nvidia-smi over SSH, then syncs code and starts the job in a named screen session with the GPU bound through CUDA_VISIBLE_DEVICES. It then confirms the session is running with screen -ls and reports back.

If you ask for Weights & Biases logging and the script lacks it, the agent adds wandb.init, wandb.log, and wandb.finish calls, then checks wandb status on the target machine. Metrics it may log include train/loss, train/lr, eval metrics and throughput.

• Pre-flight: nvidia-smi query for memory used and total per GPU
• Sync: rsync with include rules for *.py, or git push and pull
• Launch: screen session per experiment, output piped through tee to a log file
• Verify: screen -ls on the remote machine
• Notify: Feishu message only if ~/.codex/feishu.json is configured and not off

Examples of experiments the skill handles

The skill is most useful when you have several runs to place across GPUs. It can launch them in parallel on different GPUs, one screen session each, and report which GPU each run uses. For cloud use, it records the Vast.ai instance id and estimated hourly cost before launch so cleanup is traceable.

For Modal, the skill runs a modal run command after checking secrets, volumes, image dependencies, and output persistence. It records the app or function name, GPU type, timeout, and where results will be stored.

• Three seeds of one training script, each on a free GPU of a shared lab server
• A git-based sync to a server that pulls the latest committed code
• A Vast.ai run that copies logs and checkpoints, then destroys only its recorded instance
• A Modal serverless training run with mounted volumes for outputs

Safety notes: scripts and network

This skill runs scripts and uses the network. It runs SSH, rsync or git commands, and screen on the machines you configure, and it can call Vast.ai and Modal CLIs if you have credentials. It can also post to Feishu if you have that config. Review the commands it proposes before you approve them, especially on paid cloud GPUs.

Auto-destroy is the risky part. The skill is written to destroy a Vast.ai instance only after the process has exited and results are copied to a durable location. If any copy fails, it should not destroy the instance. Unless AGENTS.md explicitly says auto_destroy: true, it asks for confirmation first.

• Do not enable auto_destroy until you trust your copy destinations
• If the command fails to start or the instance is unreachable, the skill says not to relaunch blindly
• Keep API keys out of AGENTS.md where possible; the skill uses wandb login with a key you supply

Tips for getting reliable results

Keep data and checkpoints out of the sync path. The rsync default only includes Python files, so put large data on the machine itself or mount it. Confirm the conda activation line in AGENTS.md works in a plain SSH session before you rely on the skill.

Record which GPU and screen session each run uses, since the skill reports these. Use the logs written by tee to check progress rather than re-launching a run.

• Test the SSH alias and conda command manually first
• Prefer code_sync: git when you run the same code on several servers
• Use a different screen name for each experiment so sessions do not collide
• Check nvidia-smi yourself if a GPU looks busy but you expect it free