How to use the GKE TPU Performance Troubleshooter agent skill

What does the skill make the agent do?

When a TPU training job drops in speed, for example when TPU duty cycle falls by 15% or more, the skill tells the agent to investigate why. It reads ML Diagnostics monitored events and one-minute Cloud Monitoring metrics, then sorts the cause into one of two groups. The first is a workload bottleneck: memory, host memory or host CPU. The second is hardware and network throttling. The agent then suggests fixes and links to the relevant documentation sections.

The skill is designed to be read-only. The agent runs diagnostic commands, and any change to the cluster is left for you to apply.

• Workload bottlenecks: HBM capacity, host memory utilization, host CPU utilization
• Infrastructure throttling: interconnect, thermal or power throttling, memory bandwidth, and Top-of-Rack (ToR) network faults on specific instances
• Cases it hands off: full execution stalls with HANG_DETECTED logs, and pod eviction or restart problems

How do you install the GKE TPU troubleshooter skill?

You can install the skill in either of two ways. For Claude Code, download the skill's ZIP from this page and unzip it into ~/.claude/skills/ to make it available in all your projects, or into a project's .claude/skills/ folder to limit it to that project. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

After installing, the skill's folder should contain SKILL.md and a references folder with attribution.md, path-a-workload-bottlenecks.md, path-b-infrastructure-throttling.md, path-c-no-detected-analyzer.md and path-d-healthy-telemetry.md. The agent reads SKILL.md first and opens the reference files as needed.

• Personal install: unzip into ~/.claude/skills/
• Project install: unzip into the project's .claude/skills/
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP

What do you need before you start?

The skill runs real Google Cloud commands, so the environment has to be ready first. You need the Google Cloud SDK (gcloud) with the alpha component, which provides the gcloud alpha mldiagnostics commands, plus kubectl. Your project needs billing enabled and the container.googleapis.com, monitoring.googleapis.com and hypercomputecluster.googleapis.com APIs turned on.

For IAM, the skill names three roles: Cluster Director Editor, Monitoring Viewer and Kubernetes Engine Viewer. Kubernetes Engine Cluster Admin is only needed for the high-risk remediation steps. On the cluster side, Workload Monitoring is enabled by default for the jobset and job GKE job types, but it needs a GKE version that supports ML Diagnostics, as stated in the 'Configure GKE for ML Diagnostics' documentation. The skill only works with JAX on TPUs; PyTorch and other frameworks will not show ML runs or monitored events.

• Google Cloud SDK (gcloud) with the alpha component, and kubectl
• Billing enabled on the project, with GKE, Cloud Monitoring and Cluster Director APIs enabled
• Roles: Cluster Director Editor, Monitoring Viewer, Kubernetes Engine Viewer
• Kubernetes Engine Cluster Admin only for high-risk remediation
• A GKE version that supports ML Diagnostics, and JAX workloads on TPUs

How do you trigger the skill with a natural request?

You do not need to type a command name. Describe the symptom in plain words and mention TPUs, training speed or GKE. The skill's description lists the situations it matches: TPU training throughput or duty cycle dropping without crashing pods, PERFORMANCE_DEGRADATION monitored events firing, and slow multi-slice training steps.

If your problem is a full stall with HANG_DETECTED logs, or pods being evicted and restarted, the skill says to use other skills instead, named gke-ai-troubleshooting-tpu-mxla-hang and gke-ai-troubleshooting-jobset-interruption. Say so in your request so the agent picks the right one.

• 'My JAX training on our GKE TPU slice got about 20% slower after the last restart. Find out why.'
• 'Check the ML Diagnostics PERFORMANCE_DEGRADATION events for my multi-slice run.'
• 'TPU duty cycle dropped and the pods are still running. Is it memory or the network?'

How does the agent work through a diagnosis?

The skill sorts findings into three routes. Category A covers workload resource bottlenecks, where the firing analyzer points to HBM capacity, host memory or host CPU. The agent follows Path A, which is about optimizing the workload and profiling it. Path A says never to cordon, drain or replace TPU nodes for these causes, because the documented actions are workload changes. If on-demand profiling is set up, the agent can list profiler targets and capture a profile using gcloud alpha mldiagnostics commands.

Category B covers infrastructure and network fabric throttling on specific instances. Path B asks the agent to map the reported Compute Engine instance IDs to GKE node names with kubectl get nodes, and to review the analyzer's recommended actions. Node repair goes through GKE rather than Compute Engine, and the skill points to Google Cloud Support for escalation. Category C covers a PERFORMANCE_DEGRADATION event where no analyzer reports a detection, and the reference files cover that case and a healthy-telemetry path.

What are the safety notes for scripts and network access?

The skill runs scripts and uses the network. It calls Google Cloud through gcloud and kubectl, and it reads monitored events and Cloud Monitoring metrics. Its written rule is read-only: run diagnostic commands only, never drain, delete or re-create nodes, and never run any command that changes the cluster. Fixes are given to you to apply yourself.

The skill also asks the agent to tag every gcloud command it runs in the terminal with an attribution variable that names only the skill and its version. The tag is set inline on each command, not through gcloud config set, so it does not persist. Review commands before you approve them, and use an account whose roles match what you intend to allow.

• Read-only by rule; remediation is yours to apply
• High-risk steps need Kubernetes Engine Cluster Admin
• Commands are tagged inline for usage attribution, per references/attribution.md