Skip to content
Get packs

Guide · Agent skill

How to use the GKE TPU Metrics Monitoring agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

This guide explains how to install and use the gke-ai-troubleshooting-tpu-metrics-monitoring agent skill, a set of instructions that lets an AI agent check the health of Google Kubernetes Engine clusters running TPUs (Google's AI accelerator chips) using Cloud Monitoring metrics. It is aimed at engineers and platform teams who have not used agent skills before. By the end you will know how to install the skill, ask for TPU health checks in plain language, and read what the agent reports.

GKE TPU Metrics Monitoringgoogle/skills · Apache-2.0

Quick start

  1. 1Download the ZIP. Download the skill ZIP from this page and unzip it. Confirm the folder contains SKILL.md.
  2. 2Place the skill folder. Move the unzipped folder into ~/.claude/skills/ for personal use, or into the project's .claude/skills/ directory.
  3. 3Or upload it in the apps. In the Claude apps, open Settings, go to Capabilities, then Skills, and upload the skill.
  4. 4Check prerequisites. Confirm your GKE cluster runs TPUs, system metrics are enabled, the version is 1.27.4-gke.900 or later, and your identity can read Cloud Monitoring metrics.
  5. 5Ask in plain language. Describe the TPU problem, for example a slowdown or a node pool that seems unavailable, and include the project, cluster and location.
  6. 6Review the findings. Read the metrics the agent reports and compare any interruptions or unready nodes with your workload timeline.

What an agent skill is and what this one makes the agent do

An agent skill is a folder containing a SKILL.md file of instructions, plus optional reference files and scripts, that an AI agent loads when a task matches the skill's description. This skill is for diagnosing whether slowdowns or interruptions on GKE TPU workloads come from the underlying infrastructure.

Once installed, the agent follows a structured set of steps: it verifies that TPU runtime metrics are configured, reads TensorCore duty cycle and TPU memory metrics, checks node readiness and node pool status, looks at maintenance or preemption interruptions, and can calculate MTTR (mean time to recovery) and MTBI (mean time between interruptions) over the last 7 days.

  • Checks duty cycle, memory use and TensorCore utilization metrics on TPU containers and nodes
  • Checks node Ready conditions and multi-host TPU node pool availability
  • Breaks down interruptions by type and reason, including HW/SW Maintenance
  • Calculates MTTR and MTBI using PromQL queries

Install the skill from the ZIP download

Download the ZIP for this skill from this page and unzip it. The skill folder should contain SKILL.md, references/ and scripts/. For a personal install, place the folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ directory.

If you use the Claude apps instead, you can upload the skill under Settings → Capabilities → Skills.

  • Personal install: ~/.claude/skills/
  • Project install: the project's .claude/skills/ folder
  • Claude apps: Settings → Capabilities → Skills, then upload

Prepare your Google Cloud environment

The skill only works against live data, so your environment has to be ready first. The source lists these requirements: a Google Cloud project with a GKE cluster that runs TPUs, GKE system metrics enabled, GKE version 1.27.4-gke.900 or later, and, if your workload uses JAX, JAX 0.4.14 or later. Your Google Cloud identity needs permission to read Cloud Monitoring metrics.

For commands run in the terminal you also need the gcloud CLI with access to Cloud Monitoring. The skill's first step checks the Pod specification too: the TPU container should expose containerPort 8431 so Prometheus can scrape runtime metrics.

  • Google Cloud project with a TPU-enabled GKE cluster
  • GKE system metrics enabled on the cluster
  • GKE 1.27.4-gke.900 or later; JAX 0.4.14 or later if used
  • Access to read Cloud Monitoring metrics
  • gcloud CLI for terminal commands
containerPort: 8431

Trigger the skill with natural requests

You do not run the skill by name. Describe the problem in plain language and the agent will match it to the skill's description. The description covers TensorCore duty cycle, TPU memory, node readiness, multi-host node pool availability, maintenance or preemption interruptions, and MTTR or MTBI for GKE TPUs.

Give the agent the details it needs, such as your project ID, cluster name, location and, if you know them, the node or node pool name. The skill can also look these up with available GKE and Cloud tools.

  • "Is the slowdown in my TPU training job on cluster tpu-train caused by node problems?"
  • "Check whether node pool tpu-pool-1 is available and Running."
  • "Did we have any maintenance or preemption interruptions on my TPU nodes this week?"
  • "Calculate MTTR and MTBI for the TPU node pools in cluster tpu-train over the last 7 days."

What the agent checks and how to read the results

The skill maps each question to metrics and PromQL queries. For example, node readiness is read from kubernetes_io:node_status_condition, node pool status from kubernetes_io:node_pool_status, and interruptions from kubernetes_io:node_interruption_count. The reference file failure_signatures.md explains what common patterns mean. A Ready condition with status False, or a node pool with status Error, points to infrastructure. A PreemptionEvent interruption type is common with Spot VMs and means the workload needs rescheduling.

Each metric is sampled over a window (for example 60 seconds for duty cycle), so a short dip may not reflect a real outage. Compare the agent's findings with your own workload logs before drawing conclusions, because the skill does not inspect application logic.

  • Node Unready: Ready=False or Unknown means the node is unhealthy
  • Node Pool Error: status Error often means a provisioning failure
  • Preemption: PreemptionEvent interruptions
  • Host error: TerminationEvent interruptions
kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}

Safety notes: scripts and network access

This skill runs scripts and uses the network. The included scripts/validate_queries.sh file is part of the package; review it before running it. The skill's queries run against your live Google Cloud project, so the agent needs valid credentials and you should only grant access to projects you are allowed to monitor.

The skill requires that every gcloud command run in the terminal is tagged with a usage attribution environment variable that names the skill and its version. This tag is set inline on each command and is not saved with gcloud config set. The skill does not ask for your data to be included in the tag. Commands are meant for direct terminal use, not for generated scripts, Terraform or user code.

  • Read scripts/validate_queries.sh before running the skill's scripts
  • The skill needs network access to query Cloud Monitoring
  • Attribution tag is set per command, not persisted

Tips for getting useful results

Name your cluster, location and node pool up front so the agent does not have to guess. Ask narrow questions, such as one node pool over one time window, rather than asking for everything at once. For interruption counts, specify the period, because the MTTR and MTBI calculations use the last 7 days.

If results look empty, confirm that GKE system metrics are enabled and that your Pod exposes port 8431, since the metrics are not available without them.

  • Give the cluster name, location and node pool name in your request
  • Ask about one question at a time
  • Confirm system metrics and port 8431 if no data appears

What people use it for

  • Finding out whether a slow multi-host TPU training run was caused by node interruptions
  • Checking whether a TPU node pool is Running and fully available after a provisioning issue
  • Counting HW/SW Maintenance and preemption events on TPU nodes over the last 7 days
  • Reporting MTTR and MTBI for TPU node pools to a platform or SRE team

Frequently asked questions

How do I install the GKE TPU metrics monitoring skill?

Download the ZIP from this page and unzip it. Place the folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ directory. In the Claude apps, you can instead upload it under Settings → Capabilities → Skills.

What do I need before the skill works?

You need a Google Cloud project with a GKE cluster running TPUs, GKE system metrics enabled, GKE 1.27.4-gke.900 or later, and permission to read Cloud Monitoring metrics. If your workload uses JAX, it should be version 0.4.14 or later.

Is this skill free to use?

The skill is licensed under Apache-2.0. The source does not state pricing for Google Cloud Monitoring or GKE usage, so check your own Google Cloud billing for those costs.

Can I use it for GKE workloads without TPUs?

No. The skill is scoped to TPU metrics and says not to use it for general non-TPU GKE workload monitoring.

Will it help debug my application code or logs?

Not directly. The skill focuses on metric queries rather than application logs or code errors, so use it to check whether infrastructure is the cause.

Why do I get no data from the queries?

Check that GKE system metrics are enabled on the cluster, that your Pod exposes containerPort 8431, that your JAX and GKE versions meet the minimums, and that your credentials can read Cloud Monitoring.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits engineers and SRE or platform teams running TPU training or inference on GKE who need to tell infrastructure interruptions apart from other slowdowns. It is not suitable for non-TPU GKE workloads, application-level debugging, or anyone without access to the relevant Google Cloud project. Use a general observability workflow for those cases.

Download the skill