Skip to content
Get packs

Guide · Agent skill

How to use the DGX Spark Training Gotchas agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

Spark Training Gotchas is an agent skill that gives an AI agent a checklist of ten known failure modes for ML training on an NVIDIA DGX Spark, a desktop machine with a GB10 chip and 128GB of memory shared between CPU and GPU. It is for engineers whose training run fails to start, runs out of memory below the 128GB limit, or slows down partway through, and for anyone about to start a multi-hour job on a Spark. After this guide you will know how to install the skill, how to ask for it in plain language, and what it can and cannot check for you.

DGX Spark Training Gotchaswshobson/agents · MIT

Quick start

  1. 1Download the skill ZIP. Get the spark-training-gotchas ZIP from this page.
  2. 2Unzip it into a skills folder. Unzip into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder for one project. Alternatively, upload it in the Claude apps under Settings, then Capabilities, then Skills.
  3. 3Confirm your hardware and tools. Check that you are on a DGX Spark (GB10) with bash, python3, nvidia-smi and free available, and that PyTorch is installed.
  4. 4Ask in plain language. Describe the symptom or the planned job and mention DGX Spark or GB10, for example a run that OOMs while memory looks free.
  5. 5Run the preflight. Read assets/preflight.sh, then run it and read the one-line result for each gotcha, G1, G3, G4, G7 and G9.
  6. 6Apply the matching fix. Match any failing line to its G-number, check the detail in references/gotcha-checks.md, and apply the fix after manual review.

What is an agent skill and what does this one do?

An agent skill is a folder of instructions and supporting files that an AI agent can load when a request matches the skill's description. This skill's description says it is for preflight and diagnosis of the ten known failure modes for ML training on NVIDIA DGX Spark.

When you ask for help with a failing or slow run, the agent can match the symptom to one of the ten named issues, labelled G1 to G10, explain the likely cause and suggest a fix. It can also run a short preflight script that reports some checks automatically, such as the CUDA version PyTorch was built against, free memory, and GPU temperature and power draw.

  • G1: CUDA 12/13 ABI mismatch, shown as an undefined symbol import error or a segfault
  • G2: flash-attn using the wrong backend
  • G3: out-of-memory errors while nvidia-smi still shows headroom
  • G4: throughput drops or reboots from thermal and power limits
  • G5: memory-bound steps slower than the spec bandwidth suggests
  • G6: cache eviction from other GPU processes sharing memory
  • G7: NVFP4 running slower than FP8 on SM121
  • G8: official playbooks that fail outright
  • G9: environments breaking after installs, fixed by using a container
  • G10: two-Spark tensor parallel hanging, where only DDP or FSDP is advised

How do you install the Spark Training Gotchas skill?

Download the ZIP of the skill from this page. Then either unzip it into your personal skills folder at ~/.claude/skills/, or into a project's .claude/skills/ folder to make it available only in that project.

If you use the Claude apps instead of a local setup, upload the skill under Settings, then Capabilities, then Skills.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into the project's .claude/skills/ folder
  • Claude apps: Settings → Capabilities → Skills, then upload the skill

How do you trigger the skill with a natural request?

You do not need to type the skill name. The skill is meant to load when your request matches its description, so describe the symptom or the task in plain words. Mention DGX Spark or GB10 so the agent knows which hardware you mean.

Good requests name the situation: a run that will not start, a run that hits out-of-memory while memory looks free, a job that slows down after a few hours, or a plan to start a long training job. Include the error text if you have it.

  • Why does my PyTorch training on the DGX Spark fail with an undefined symbol error?
  • My QLoRA run on the Spark runs out of memory even though nvidia-smi shows free memory. Can you check the gotchas?
  • Before I start a 12-hour fine-tune on the GB10, run the Spark preflight checks.
  • My training slows down after two hours on the DGX Spark. What could cause that?
  • I want to train across two DGX Sparks. Is tensor parallelism safe?

What does the preflight script check?

The skill includes a preflight script, assets/preflight.sh. It runs the checks for G1, G3, G4, G7 and G9 and prints one line per gotcha. Each line starts with the G-number, followed by PASS, FAIL or WARN where the check can be automated, SKIP when a check is unavailable, or INFO: for a raw reading, which is the case for G3 and G4.

You can also run three quick commands yourself. The first reports the CUDA version PyTorch was built against, which should show 13.x for G1. The second reports GPU capability, which should show (12, 1) for G7. The third looks for container markers for G9. The source notes that NGC container builds have no +cu130 tag, so its absence is not a failure on those containers.

python3 -c "import torch; print(torch.version.cuda)" python3 -c "import torch; print(torch.cuda.get_device_capability())" { [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown

How do you read the results and act on them?

Match the failing line to its gotcha, then follow that gotcha's fix. The skill's detailed commands for each check are in references/gotcha-checks.md. For example, for G3 the source says to read free -g and /proc/meminfo rather than nvidia-smi, because nvidia-smi can report free memory that does not reflect what the run can use.

Do not apply fixes blindly. The source says some checks need manual review, including flash-attn presence, running processes, upstream playbook issues and parallelism configuration. The script also does not verify the kernel target architecture.

  • G3 fix: drop the page cache with sync; echo 3 > /proc/sys/vm/drop_caches. This needs root and should be done between runs, not during training.
  • G4: if power plateaus under 240W while temperature climbs, treat throttling as the likely cause and improve cooling or cap run length.
  • G7: stay on FP8 unless the build targets sm_121a.
  • G10: on a two-Spark setup, choose DDP or FSDP and never tensor parallelism.

What does the skill need, and what are the safety notes?

The checks are only meaningful on NVIDIA DGX Spark (GB10) hardware. The machine needs bash and python3. PyTorch must be installed for the G1 and G7 checks. The system also needs nvidia-smi and free.

The skill runs scripts, so read assets/preflight.sh before running it. The skill does not use the network. The source describes the checks as read-only unless noted, but the drop-caches fix for G3 changes system state and needs root, so run it deliberately.

  • Hardware: NVIDIA DGX Spark (GB10)
  • Tools: bash, python3, nvidia-smi, free
  • Python package: PyTorch (torch), for G1 and G7
  • Scripts: yes, assets/preflight.sh
  • Network: not used

What tips help you get the most from it?

Run the checks before a long job, not after it has failed. The skill's own advice is to read it before a long run. Keep the environment consistent: the source advises a container, such as an NGC container, over bare pip, because pip can let packages drift apart. Finally, measure your own unit under load before making hardware or cooling decisions from the thermal and power notes.

  • Run the preflight before any multi-hour or multi-epoch job
  • Prefer a container over bare pip for a stable environment
  • Use the 180 to 192 GB/s measured bandwidth range, not the 273 GB/s spec, when budgeting throughput
  • Do not stack a large uncapped job with other GPU-resident servers

What people use it for

  • Checking a PyTorch install on a DGX Spark for the CUDA 13 mismatch that causes undefined symbol errors (G1).
  • Diagnosing an out-of-memory crash during QLoRA loading on a Spark where memory appears free (G3).
  • Planning a multi-hour fine-tune by checking for power-capped throughput before starting (G4).
  • Deciding whether to wire two Sparks together and which parallelism strategy to use (G10).

Frequently asked questions

How do I install the spark-training-gotchas agent skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps you can instead upload it under Settings, then Capabilities, then Skills.

What hardware do I need to use this skill?

The checks are meaningful only on an NVIDIA DGX Spark with a GB10 chip. You also need bash, python3, nvidia-smi and free on the system, and PyTorch installed for the G1 and G7 checks.

Is the Spark Training Gotchas skill free to use?

The item is listed under the MIT licence. Check the licence file in the source repository for the full terms.

Why does my DGX Spark run run out of memory when nvidia-smi shows free memory?

This matches gotcha G3, where unified memory accounting can differ from what nvidia-smi reports. The skill says to read free -g and /proc/meminfo instead. Dropping the page cache between runs is one suggested fix, and it needs root.

Does this skill work on ordinary GPUs or cloud instances?

No. The skill covers GB10-specific gotchas, so these checks do not apply to ordinary GPUs, cloud instances or non-Spark hardware.

Can the agent fix my training run automatically?

Not fully. The skill maps symptoms to named issues and suggests fixes, but some checks, such as flash-attn presence, running processes, upstream playbook issues and parallelism config, need manual review, and the script does not verify kernel target architecture.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you run ML training on an NVIDIA DGX Spark and want a named checklist and a preflight script before long or failing runs. Pick something else if you train on ordinary GPUs, cloud instances or other hardware, since these GB10-specific gotchas will not apply, or if you want fixes applied without manual review.

Download the skill