Skip to content
Get packs

Training Health Check

Periodically checks WandB training metrics to catch NaN, divergence, or stalled runs early

Source Download skill
Repository
wanshuiyin/Auto-claude-code-research-in-sleep
Stars
18,482
Licence
MIT
Files
1 file

What it does

When installed, the agent can monitor a machine-learning training run on a schedule. It reads metrics from Weights & Biases (WandB), a popular experiment-tracking service, or falls back to reading the training log over SSH. It looks for warning signs such as NaN values, loss that keeps rising, or eval scores that fall well below baseline. If a run looks clearly broken it can stop the training session and save the run link and metrics as evidence. Unclear cases can be sent to a second AI reviewer (Codex) for a one-off opinion. Process-level checks such as crashes or idle GPUs are handled by a separate tool, watchdog.py.

Great for

  • ML researchers running long GPU training jobs who want to avoid wasting compute on broken runs
  • Engineers who already log training to WandB and want automated periodic health checks

Skip it if

  • Users without a WandB account or SSH access to their training log, since the skill has no other data source (the process would only report connectivity issues)
  • Anyone who wants the agent to stop training automatically without review: the skill can kill a session on its own judgment, and metric thresholds are heuristic, so a false positive could end a valid run

Try asking

  • “Check the WandB run myteam/llm-pretrain/abc123 for NaN loss or divergence and tell me if it should be stopped”
  • “Set up periodic health checks on my current training run and back off the interval while it stays healthy”
  • “Read the last 100 lines of the training log on my server and summarise whether the loss is still decreasing”
Step-by-step guide: how to use Training Health CheckInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 1 original files, the MIT licence and an attribution note.

    training-check.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip training-check.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip training-check.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the full WandB run path as the argument so the agent does not have to guess the entity and project
  • Keep the default cautious rules: judge trends over several checkpoints rather than single spikes, and check that the reviewer model name is valid for your account before relying on it
  • Pair it with watchdog.py so crashes and idle GPUs are caught immediately while this skill watches training quality

What's inside

  • SKILL.md

Redistributed unmodified from wanshuiyin/Auto-claude-code-research-in-sleep under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.