Guide · Agent skill
How to use the Spark Memory & Thermal Ops agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
Spark Memory & Thermal Ops is an agent skill for machine learning training on NVIDIA DGX Spark, which uses a GB10 chip with one shared 128GB memory pool. It is written for engineers fine-tuning large models who hit out-of-memory errors or see slowdowns in multi-hour jobs. After this guide you will know how to install the skill, how to phrase requests so the agent uses it, and what it will and won't do on your machine.
Spark Memory & Thermal Opswshobson/agents · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it so the spark-memory-thermal-ops folder contains SKILL.md.
- 2Place it where your agent looks. Move the folder to ~/.claude/skills/ for personal use or to the project's .claude/skills/ folder, or upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
- 3Confirm requirements. Check that you have DGX Spark or GB10 hardware, nvidia-smi, the NVIDIA driver, free and awk.
- 4Ask a specific planning question. Describe the model, method, batch or packing length and hardware so the agent can size the run against free memory.
- 5Start monitoring for long runs. Run bash assets/thermal-sample.sh 30 thermal.log alongside your training log to correlate slowdowns with thermal readings.
- 6Work any OOM in order. If a job runs out of memory, follow the ladder: flush the cache between runs, then shrink packing or batch, then downgrade the method.
What an agent skill is and what this one makes the agent do
An agent skill is a folder with a SKILL.md file that tells an AI agent when and how to apply a specific body of knowledge. This skill covers three jobs: planning whether a training run fits in the shared 128GB pool, working through a fixed ladder of fixes when a job runs out of memory, and interpreting temperature and power during long jobs.
On DGX Spark, the skill points out that nvidia-smi can underreport memory pressure, so the agent budgets against free memory from free -g instead. It also treats a sustained power draw near 100W as the platform's normal cap rather than a configuration bug.
- Estimate memory headroom before a run, using a worksheet in references/uma-accounting.md
- Work an out-of-memory error in order: flush the buffer cache, reduce batch or packing length, then downgrade the method
- Read temperature and power logs to tell thermal slowdowns from configuration problems
- Start a background sampler that writes readings to a CSV file
Install the skill from the ZIP
Download the ZIP for this skill from this page and unzip it. You can install it for yourself or for one project. For all your projects, place the unzipped folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.
If you use the Claude apps instead of a local setup, upload the ZIP under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/
- Project install: the .claude/skills/ folder inside your project
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Check that your machine meets the requirements
The skill is built for NVIDIA DGX Spark or GB10 hardware. It needs nvidia-smi and the NVIDIA driver, plus the Linux tools free and awk. It also refers to the separate spark-training-gotchas skill for some diagnostics, so install that too if you want those checks.
The buffer cache flush step needs root access. If you cannot safely run commands as root, skip that step and treat the rest of the ladder as your guide.
- NVIDIA DGX Spark or GB10 hardware
- nvidia-smi and the NVIDIA driver
- free and awk available on the command line
- Root access for the buffer cache flush step only
- spark-training-gotchas for launch-time failures
Trigger the skill with natural requests
You don't need a special command. The agent loads the skill when your request matches its description. Plain requests work well, especially when they name the model, the method, the batch or packing length, and the symptom.
Good triggers describe a planning question, an out-of-memory error, or a long-run monitoring question. Mention the hardware too, since the skill is specific to DGX Spark.
- Will a 70B QLoRA run fit on my DGX Spark with packing length 4096?
- My training job OOMs on unified memory during model load. What should I try first?
- I'm starting a 6-hour fine-tune on GB10. How should I monitor temperature and power?
- Can I run vLLM and a training job on the same DGX Spark?
Use the memory and thermal commands the skill references
The skill shows the commands it relies on. To check free memory in gigabytes, the agent uses free -g. To check for processes that may hold GPU memory, it uses ps. To log temperature and power, it runs the bundled sampler, which takes an interval in seconds and an output file.
You can run these yourself to see what the agent sees. The sampler writes CSV output so its timestamps line up with your training log.
free -g | awk 'NR==2 {print "free:", $4, "GB"}'
ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep
bash assets/thermal-sample.sh 30 thermal.logWalk through the out-of-memory ladder
When a job OOMs on unified memory, the skill asks the agent to try fixes in a fixed order, because each later step is more disruptive. Reducing batch size is never the first step. The buffer cache flush is the first step, and it only applies between runs because it needs root.
If the flush does not free enough memory, shrink packing length first, since it drives activation memory more directly at long context. If that also fails, drop from QLoRA to bf16 LoRA, since QLoRA's dequantization buffers can OOM even though its steady-state footprint is smaller. A QLoRA OOM does not prove the model won't fit.
- Step 1: flush the buffer cache (needs root, between runs)
- Step 2: reduce batch size or packing length
- Step 3: downgrade the method from QLoRA to bf16 LoRA
- Beyond these: a smaller model or multi-Spark setups
sync; echo 3 > /proc/sys/vm/drop_cachesRun a trainer and an inference server together safely
The 128GB pool is shared, so an uncapped trainer and an uncapped inference server such as vLLM or Ollama can evict each other's memory without either log showing an OOM. The skill's guidance is to run one heavy job at a time. A small, capped workload can coexist, for example a LoRA fine-tune under 4GB alongside vLLM capped at gpu-memory-utilization<=0.5.
Check the other process's cap, not just whether it is running, and stop unrelated uncapped servers before a long or full-pool run.
What people use it for
- Checking whether a 70B QLoRA run fits in the shared pool before launch, using the worksheet's anchors
- Diagnosing an out-of-memory error that happens during model load rather than during training
- Logging temperature and power every 30 seconds to see if a multi-hour run's slowdown is thermal
- Deciding whether a trainer can share the machine with a capped vLLM server
Frequently asked questions
How do I install the spark-memory-thermal-ops skill?
Download the ZIP from this page and unzip it. Put the folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder, or upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
What hardware does this skill require?
It is built for NVIDIA DGX Spark or GB10 hardware. It also needs nvidia-smi, the NVIDIA driver, and the Linux tools free and awk. The skill says it does not apply to other machines with discrete GPU memory.
Is the Spark Memory & Thermal Ops skill free?
The skill is published under the MIT licence, as listed on this page. Check the licence file in the source repository for the full terms.
Does the skill need internet access?
No. The listing states it does not use the network. It does run scripts, including the bundled thermal sampler, so review them before use.
Why does my job OOM even though nvidia-smi shows free memory?
On DGX Spark, nvidia-smi reports only CUDA-allocator-visible memory. Page cache and mmap'd pages use the same unified pool, so headroom can look available and still run out. The skill recommends budgeting against free -g instead.
Is the skill the same as spark-training-gotchas?
No. This skill covers memory planning, OOM handling and thermals during a running job. Launch-time failures such as library version mismatches or flash-attn problems belong to the separate spark-training-gotchas skill.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill is a good fit if you fine-tune large models on a DGX Spark or GB10 and need a disciplined approach to out-of-memory errors and long-run thermals. It is not useful on other GPU machines, and it won't fix launch-time library errors, for which you should use spark-training-gotchas.
Download the skill