Skip to content
Get packs

DGX Spark Training Gotchas

Preflight-check ten known ML training failure modes on NVIDIA DGX Spark (GB10) before long runs.

Source Download skill
Repository
wshobson/agents
Stars
42,095
Licence
MIT
Files
3 files

What it does

This skill gives an AI agent a checklist of ten known problems that cause ML training to fail, crash, or slow down on an NVIDIA DGX Spark (a desktop machine with a GB10 chip and 128GB of memory shared between CPU and GPU). When you ask for help with a failing or slow training run, the agent can match the symptom to one of the ten named issues (G1 to G10), explain the likely cause, and suggest a fix. It can also run a short preflight script that reports some of the checks automatically, such as the CUDA version torch was built against, free memory, and GPU temperature and power draw.

Great for

  • Engineers starting a multi-hour or multi-epoch fine-tuning job on a DGX Spark
  • Anyone debugging an import error, segfault, or out-of-memory error on a Spark even though nvidia-smi shows free memory
  • Teams deciding whether to wire two Sparks together for distributed training

Skip it if

  • Training on ordinary GPUs, cloud instances, or non-Spark hardware, where these GB10-specific gotchas do not apply
  • Fixing a run blindly from the checklist alone: some checks (flash-attn presence, running processes, upstream playbook issues, parallelism config) need manual review, and the script does not verify kernel target architecture
  • Making hardware or cooling decisions from the thermal and power notes without measuring your own unit under load

Try asking

  • “My training run on DGX Spark throws 'undefined symbol' on the first .cuda() call. Which gotcha is this?”
  • “Run the preflight checks before I start my 12-hour fine-tune on the Spark.”
  • “My job runs out of memory even though nvidia-smi shows free memory. What should I check?”
  • “Should I use FP8 or NVFP4 for this inference job on my DGX Spark?”
Step-by-step guide: how to use DGX Spark Training GotchasInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    spark-training-gotchas.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip spark-training-gotchas.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip spark-training-gotchas.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run the preflight script before any long job, and read the G3 and G4 INFO lines yourself, since they are raw readings that need judgment.
  • Treat the G7 result as hardware capability only; separately confirm your kernel build targets sm_121a before choosing NVFP4.
  • For G8, check the NVIDIA dgx-spark-playbooks issues page yourself before trusting a recipe for an expensive run.

What's inside

  • assets/preflight.sh
  • references/gotcha-checks.md
  • SKILL.md

Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.