How to use the Experiment Queue for GPU Batches agent skill

What is an agent skill and what does this one do?

An agent skill is a folder with a SKILL.md file and optional scripts that an agent loads when a request matches its description. This skill, named experiment-queue, is an SSH job queue for multi-seed and multi-config ML experiments.

Once active, the agent can take a set of jobs, start them on free GPUs on your remote server, retry jobs that run out of GPU memory after a delay, clean up leftover terminal sessions, and launch the next group of jobs only after the previous group has finished. It also keeps a state file so you can check progress overnight, showing how many jobs are done, running, and pending.

• Starts jobs on GPUs that are free, up to a configured max_parallel limit.
• Retries jobs that fail with out-of-memory (OOM) errors after a delay, up to a set number of attempts.
• Parks jobs that fail for other reasons as stuck, for a human to review.
• Cleans up stale screen sessions whose Python process has already exited.
• Starts the next wave only after the current wave has exited and GPU memory has dropped below a threshold.
• Starts a student job only after its teacher checkpoint exists (preconditions).

What do you need before you start?

The skill assumes your remote environment is already set up. It does not install your project for you.

• SSH access to a remote GPU server, with screen installed.
• Python 3 on the remote host.
• A conda environment on the remote host with your project's dependencies.
• Claude Code or another agent that can run Bash commands.
• The related skills run-experiment and monitor-experiment, which the skill references in its allowed tools.
• Your project's jobs described either as a YAML manifest, a grid specification, or a plain-language description the agent turns into a manifest.

How do you install the experiment-queue skill?

Download the ZIP for this skill from our page and unzip it. Put the resulting experiment-queue folder in one of two places. For a personal install available in all projects, use ~/.claude/skills/. For a single project, use the .claude/skills/ folder inside that project.

If you use the Claude apps instead of Claude Code, you can upload the skill under Settings → Capabilities → Skills.

• Personal install: unzip into ~/.claude/skills/
• Project install: unzip into the project's .claude/skills/ folder
• Claude apps: upload under Settings → Capabilities → Skills

How do you trigger the skill with a natural request?

You do not need to type a command. The skill's description lists phrases that activate it, so you can ask in plain language. Mention a batch, a grid, a multi-seed sweep, or chaining experiments automatically.

Also mention the number of jobs. The skill is meant for cases where /run-experiment is not enough, which the description puts at 10 or more jobs needing orchestration.

• "Run a batch of experiments for seeds 200 to 220 across the four model sizes on my GPU server."
• "Set up a grid: N = 64, 128, 256 and n = 50K, 150K, 500K, then run it on gpu-server."
• "Auto-chain the student training after the teacher checkpoints finish."
• "Run this multi-seed sweep and tell me overnight how many jobs are done."

How does a job manifest work?

A manifest is a YAML file that lists the project, the working directory, the conda environment, the SSH host, a default command, the GPUs to use, the parallel limit, the OOM retry policy, and each job with its arguments. The agent builds or reads this file and saves a copy so the run can be reproduced.

Key fields include gpus, max_parallel, gpu_free_threshold_mib (default 500), and oom_retry with a delay and max_attempts. Each job has an id and args. Preconditions such as checkpoint_exists stop a job from starting until its required file is present.

• project, cwd, conda, and ssh identify where and how jobs run.
• default_cmd is the command each job runs, with per-job args added.
• gpus and max_parallel control how many jobs run at once.
• oom_retry sets the delay in seconds and the number of attempts after an out-of-memory failure.
• preconditions block a job until a checkpoint file exists.

What are the safety notes for running scripts and network access?

This skill runs scripts and uses the network. Its scripts are build_manifest.py, queue_manager.py, and SKILL.md. The scheduler runs detached on the server and polls GPU state on a fixed interval, so it keeps working after your terminal session closes.

Do not wrap the skill in a second loop or scheduled poll. The skill already runs its own scheduler, and an extra poll would race its wave-transition logic. Use its status output to check progress instead.

• Review the scripts before installing, since they run commands on your remote server over SSH.
• The skill uses Bash, so it needs permission to run commands.
• Do not add an external /loop or cron-style poll on top of the built-in scheduler.
• Expect failed jobs to be marked stuck rather than retried, so check them yourself.

What tips help the queue run smoothly?

Set the GPU free threshold to match your server. The default is 500 MiB. Raise it on a shared server where other users hold memory, and lower it for tight packing on a dedicated machine.

Make sure the conda environment is built and tested on the remote host before you queue jobs. If a whole wave dies at import time, the environment is usually the cause, not the queue.

• Check the status output in the morning rather than polling the server manually.
• Use a grid spec for regular combinations and a manifest for irregular job lists.
• Keep the saved manifest in the run folder so you can rerun the exact same batch.