Skip to content
Get packs

GKE TPU Performance Troubleshooter

Diagnose slow or dropping TPU training throughput on GKE using ML Diagnostics and Cloud Monitoring data

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
6 files

What it does

When installed, the agent can investigate why a Cloud TPU training job on Google Kubernetes Engine is running slower, such as a drop in TPU duty cycle of 15% or more, without the pods crashing. It reads ML Diagnostics monitored events and one-minute Cloud Monitoring metrics, then sorts the cause into either a workload bottleneck (memory, host memory or host CPU) or hardware and network throttling, and suggests fixes with links to the relevant docs. It is designed to be read-only: it runs diagnostic commands and leaves any changes to the user.

Great for

  • ML engineers running JAX training jobs on GKE TPU slices who see step times regress
  • Platform or cluster operators triaging slow multi-slice training runs
  • Teams investigating PERFORMANCE_DEGRADATION events from ML Diagnostics

Skip it if

  • Workloads using PyTorch or other frameworks, because ML Diagnostics only supports JAX on TPUs and will not list their events
  • Full execution stalls with HANG_DETECTED logs or pod eviction and restart issues, which the skill says to route to other skills instead
  • Clusters without ML Diagnostics or the required GKE version, since the analyzers will not produce data

Try asking

  • “My multi-slice TPU training throughput dropped about 20% overnight but pods are still running. Can you find out why?”
  • “Check the ML Diagnostics monitored events for my jobset and tell me whether this is a workload bottleneck or a hardware issue.”
  • “Look at the TPU duty cycle and host CPU metrics for my GKE training job and explain the step-time regression.”
Step-by-step guide: how to use GKE TPU Performance TroubleshooterInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 6 original files, the Apache-2.0 licence and an attribution note.

    gke-ai-troubleshooting-tpu-performance-degradation.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-tpu-performance-degradation.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-ai-troubleshooting-tpu-performance-degradation.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Make sure the agent has read the attribution file first, since the skill requires it before any gcloud command, and that the gcloud alpha component is installed.
  • Give the agent your project ID, cluster name and job name up front so it can run the read-only queries without guessing.
  • Review any remediation steps yourself before applying them, because the skill is written to hand fixes to you rather than run them, and high-risk steps need Cluster Admin rights.

What's inside

  • references/attribution.md
  • references/path-a-workload-bottlenecks.md
  • references/path-b-infrastructure-throttling.md
  • references/path-c-no-detected-analyzer.md
  • references/path-d-healthy-telemetry.md
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.