GKE TPU Multi-Slice Hang Troubleshooting
Diagnose frozen multi-slice TPU training jobs on GKE using MXLA hang reports and latency metrics
- Repository
- google/skills
- Stars
- 21,782
- Licence
- Apache-2.0
- Files
- 7 files
What it does
When a large AI training job spread across several Google Cloud TPU slices on Kubernetes (GKE) freezes without making progress, this skill guides the agent through a structured investigation. It reads Megascale HANG_DETECTED logs, the ML Diagnostics hang analyzer report, and multi-slice latency metrics, then sorts the likely cause into categories such as compiler divergence, data input stalls, or hardware and network faults. The agent is told to run only read-only checks and to give remediation steps to the user rather than changing the cluster itself.
Great for
- Engineers running JAX training jobs on multi-slice Cloud TPU clusters on GKE who see a job stop advancing
- Platform or ML infrastructure teams triaging recurring HANG_DETECTED events
- Anyone who needs a documented, read-only first pass before escalating to Google Cloud support
Skip it if
- Gradual slowdowns in step time without any hang, which the skill itself says belongs to a different skill; the agent may misroute or miss the cause
- Pod preemption or eviction restarts, which another skill covers
- Workloads using PyTorch or other frameworks, since ML Diagnostics only supports JAX on TPUs and will not list their runs or events
Try asking
- “My multi-slice TPU training job on GKE stopped logging steps and shows HANG_DETECTED. What is the likely cause?”
- “Run the MXLA Hang Analyzer on my jobset and tell me whether this is a compiler divergence or a hardware fault.”
- “Check the multislice latency metrics for my TPU job over the last hour and see if one slice is the outlier.”
- “Help me find which GKE TPU nodes are involved in the hang and what the documentation recommends next.”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 7 original files, the Apache-2.0 licence and an attribution note.
gke-ai-troubleshooting-tpu-mxla-hang.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-tpu-mxla-hang.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip gke-ai-troubleshooting-tpu-mxla-hang.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the agent the job name, project ID, and namespace up front so it can run the Cloud Logging, PromQL, and kubectl queries without guessing.
- Ask for the analyzer code and the linked documentation section, then decide yourself whether to apply any fix, since the skill reserves node changes for you.
- Run the skill in a project and account with read-only roles where possible, and review every gcloud command the agent proposes before it runs.
What's inside
- references/attribution.md
- references/path-a-compiler-hlo-divergence.md
- references/path-b-program-queueing-input-stall.md
- references/path-c-hardware-network-faults.md
- references/path-d-no-detected-or-unknown.md
- references/path-e-healthy-telemetry.md
- SKILL.md
Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.