Guide · Agent skill
How to use the Training Health Monitor agent skill (training-check)
Updated · written with AI from the project's own documentation and checked against our catalogue data
The training-check skill, also called the Training Health Monitor, lets an AI agent watch a machine-learning training run in an open terminal and judge whether the run is still worth continuing. It is aimed at ML engineers and researchers who run long training jobs and want quality checks on loss and eval curves, not just proof that a process is alive. After reading this guide you will know how to install the skill, how to start a monitoring session, what each report contains, and what safety limits to set before letting an agent stop a job.
Training Health Monitorwanshuiyin/Auto-claude-code-research-in-sleep · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it so the folder contains SKILL.md.
- 2Place the folder. Put it in ~/.claude/skills/ for personal use, in .claude/skills/ within a project, or upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Gather monitoring context. Collect the WandB run path or URL, fallback log path or SSH command, expected baseline, success metrics, and how the job was launched.
- 4Start the monitoring request. Ask the agent in plain words to monitor your training run, including the run reference or a short brief.
- 5Review each report. Read the data source, anomalies and evidence in each 30-minute report before trusting the decision.
- 6Confirm STOP decisions. When the agent reports STOP, check the saved evidence and FINAL_SUMMARY, since the skill may halt a job from partial logs.
What is an agent skill, and what does training-check make the agent do?
An agent skill is a folder with a SKILL.md file that gives an AI agent extra instructions for a specific task. When you ask for something that matches the skill, the agent follows its steps. This skill is named training-check, and its description is: interactively monitor training metrics from the current session, periodically checking WandB or fallback logs for NaN, divergence, plateaus, and broken runs.
Once active, the agent keeps the session open and runs a training health check roughly every 30 minutes. Each round it reads metrics from Weights & Biases (WandB) first, or falls back to log files if WandB is unreachable, and writes a short report ending in a CONTINUE, WAIT, or STOP decision.
- Looks at training loss trend, eval metrics compared with baseline, and learning rate behaviour.
- Checks for NaN or Inf in loss, gradients, activations, or logged metrics.
- Flags sudden loss spikes, divergence, repeated failed evaluations, and plateaus.
- Records evidence (WandB URL, log lines, metric values, files inspected) in each report.
Install the training-check skill
Download the ZIP for this skill from this page and unzip it. The skill folder should contain SKILL.md. Then place it in one of these locations:
- Personal skills folder: ~/.claude/skills/ (available across your projects).
- Project skills folder: .claude/skills/ inside your project (available only to that project).
- Claude apps: upload the skill under Settings → Capabilities → Skills.
Prepare the monitoring details before the first check
The skill asks for minimum context before the first check. Have these ready so the agent does not have to guess:
- WandB run path or URL, if you use WandB.
- A fallback log path, SSH command, or local command for reading recent training logs.
- The training target, expected baseline, and the metrics that define success.
- How the training was launched, so the agent knows how to stop it.
- A project notes path where decisions and evidence will be recorded.
Trigger the skill with a natural request
You do not need a special command. Describe the job and ask the agent to monitor it. The skill also accepts an optional argument hint, [wandb-run-or-monitoring-brief], which you can include as a short brief or run reference.
Read a training-check report
Each round produces one report in the terminal. The decision tells you what the agent will do next:
- CONTINUE: the run looks healthy enough; the agent checks again in 30 minutes.
- WAIT: evidence is inconclusive, too early, noisy, or temporarily unreachable. Training is not stopped.
- STOP: training is clearly problematic or no longer worth continuing. The agent stops the task, saves evidence and notes, writes a FINAL_SUMMARY, and ends monitoring.
## Training Check - <local timestamp>
- Data source: wandb_ok | log_fallback | unreachable
- Run: <wandb run or training identifier>
- Recent metrics: <loss/eval/lr/grad summary>
- Anomalies: <NaN/Inf/spike/divergence/plateau findings>
- Evidence: <WandB URL, log lines, metric values, or files inspected>
- Decision: CONTINUE | WAIT | STOP
- Reason: <why this decision is justified>
- Next check: <local timestamp, normally 30 minutes later unless ending>Safety notes: scripts, network and stopping jobs
This skill runs scripts and uses the network. Its tool permissions include Bash, and it may read logs over SSH or query WandB. Review the SKILL.md and the commands you allow before you run it on a shared or production machine.
A STOP decision means the agent will stop your training task. The skill says the agent may halt training on its own judgement from partial logs, so keep a human in the loop for expensive jobs. The skill itself says to never stop on the first sign of ordinary metric noise, and to look for sustained trends, hard failures, or clear divergence.
- If a stop_command is in the context, the agent runs it first; otherwise it picks a stop action based on how the job was launched.
- If WandB and fallback logs are both unreachable, the agent reports the connectivity issue and waits. It does not treat missing data as bad training.
- Process checks such as whether a tmux session or GPU is alive are not this skill's job; watchdog-style tools handle those better.
Tips for getting useful checks
The quality of each report depends on the context you give. A stated baseline and clear success metrics let the agent judge eval results against something concrete.
- Give a WandB run path or URL, or a log path the agent can read, rather than a vague description.
- State the expected baseline and which metrics matter most.
- Tell the agent exactly how the job was launched so a stop is possible.
- Ask for a project notes path so decisions are saved for later review.
- If a run is suspicious but not yet stop-worthy, the skill can use a shorter interval; ask for that explicitly.
What people use it for
- Watching loss and eval curves on a multi-day fine-tuning run logged to WandB.
- Catching NaN or Inf values in loss or gradients before a run wastes more compute.
- Keeping a timestamped written record of run health for a research team's decisions.
- Checking a run whose logs are on a remote machine reachable by SSH, when the agent has access.
Frequently asked questions
Is the training-check skill free to use?
This page lists the skill under the MIT licence, which the source metadata gives. Check the repository's licence file for the full terms. The source does not state any price.
What do I need to run the training-check skill?
You need Bash access to read logs or run SSH commands, WandB access (a run path or URL) if you use WandB, access to the training log files or SSH host, and a way to stop the training job, such as a tmux session, process, or scheduler command.
Does training-check work without WandB?
Yes. If WandB is unreachable or not configured, the skill reads fallback log files. If both sources are unreachable, it reports the connectivity issue and classifies the round as WAIT.
Will the agent stop my training automatically?
It can. A STOP decision makes the agent stop the training task. The skill notes that this may be based on partial logs, so review expensive runs yourself before relying on an automatic stop.
How is training-check different from a watchdog or process monitor?
Watchdog-style tools check whether processes, tmux sessions, or GPUs are alive. Training-check focuses on whether the run is still worth continuing, based on metric quality such as NaN, divergence and plateaus.
How often does the skill check my run?
The default interval is every 30 minutes. The agent announces the next check time after each report and can use a shorter interval for suspicious runs that are not yet stop-worthy.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use training-check if you run long training jobs, log to WandB or readable log files, and want an agent to watch loss and eval quality with a consistent written record. Pick something else if you only need to know whether a process or GPU is alive, or if you cannot allow an agent to stop costly jobs without a person reviewing the decision.
Download the skill