Guide · Agent skill
How to use the GKE unresponsive TPU/GPU node troubleshooting skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
This page explains how to install and use an agent skill that helps an AI coding agent investigate a Google Kubernetes Engine (GKE) node running TPUs or GPUs that has stopped responding. It is written for platform engineers, ML engineers and on-call operators who have not used agent skills before. By the end you will know what the skill does, how to add it to your agent, how to phrase a request that triggers it, and what it needs before it runs.
Troubleshoot unresponsive GKE TPU/GPU nodesgoogle/skills · Apache-2.0 Download the skill Quick start
- 1Download and unzip the skill. Download the skill ZIP from this page and unzip it so the folder contains SKILL.md and references/attribution.md.
- 2Place it where your agent can find it. Unzip into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder for one project. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Set up Google Cloud access. Install gcloud and kubectl, log in, set your project and confirm the container, compute, logging and monitoring APIs are enabled.
- 4Confirm IAM roles. Make sure you have the Kubernetes Engine Viewer, Compute Viewer, Logs Viewer and Monitoring Viewer roles.
- 5Describe the symptom. Ask the agent in plain language, giving the project, cluster, node pool, node name and incident time in UTC.
- 6Review the findings. Check whether the agent confirmed NodeStatusUnknown or ruled out a node timeout, and read the linked documentation sections.
- 7Apply any fix yourself. If a fix is suggested, review it and apply it yourself, since the skill never changes your cluster.
What the skill makes the agent do
When a GKE node with TPU or GPU hardware stops responding, its Ready condition can change to Unknown with the reason NodeStatusUnknown and the message "Kubelet stopped posting node status." This often happens after a host kernel panic, a hardware lockup, or when node auto-repair is turned off on the node pool. Pods on such a node can then stay in Terminating, which can block multi-host JobSet workloads from recovering.
The skill gives the agent a fixed diagnostic order. It sets a 60-minute investigation window around the incident time, checks the node's Ready condition, queries Cloud Logging and Cloud Monitoring in read-only mode, and then inspects serial port output for signs of a kernel panic. If the node is actually healthy, the agent is told to rule out a node timeout and move on to workload-level troubleshooting.
- Confirms the NodeStatusUnknown timeout from node conditions, logs and metrics
- Checks serial port output for a kernel panic
- Points you to likely causes and to the relevant Google documentation sections
- Never changes the cluster: any fix is handed back to you to apply
Install the skill (ZIP download or Claude app upload)
Download the skill ZIP from this page and unzip it. The skill folder contains SKILL.md and a references/attribution.md file. Keep that folder structure intact.
For personal use with Claude Code, unzip the skill into ~/.claude/skills/ so it is available in all your projects. To limit it to one project, unzip it into the .claude/skills/ folder inside that project instead.
In the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.
Install the required tools and check your Google Cloud setup
The skill expects the Google Cloud SDK (gcloud) and kubectl to be installed. The skill's prerequisites section tells the agent to verify that an active billing account is linked to your project, that you are logged in, and that the target project is set. You can ask the agent to run those checks, or run them yourself.
The required APIs are container, compute, logging and monitoring (googleapis.com). Read-only diagnosis needs the Kubernetes Engine Viewer, Compute Viewer, Logs Viewer and Monitoring Viewer roles. The Kubernetes Engine Cluster Admin role is only needed for remediation steps that you perform yourself.
- Google Cloud SDK (gcloud)
- kubectl
- Active billing account linked to the project
- IAM roles: Kubernetes Engine Viewer, Compute Viewer, Logs Viewer, Monitoring Viewer
- Kubernetes Engine Cluster Admin only for remediation you run yourself
Trigger the skill with a natural request
You do not need to type the skill name. Describe the symptom in plain language and include the details the skill asks for: project, cluster, location, node pool, node name and the incident time in RFC3339 UTC. The skill description says it should be used when nodes stop heartbeating beyond the node auto-repair threshold and pods remain stuck in Terminating.
If you do not know every value, give what you have. The skill's first step collects the parameters and builds the investigation window from the incident time.
- "My TPU node in node pool tpu-pool on cluster train-cluster is NotReady since 2025-01-10T14:00:00Z. Pods are stuck in Terminating. Can you find out why?"
- "Check whether node auto-repair is enabled on our GPU node pool and whether this unresponsive node timed out."
- "Pods on an accelerator node pool are stuck in Terminating. Confirm whether the node is NodeStatusUnknown."
Example: a read-only investigation
The agent first checks the node's Ready condition with kubectl-based inspection described in the NotReady troubleshooting docs. It then runs a Cloud Logging query filtered to k8s_node and k8s_cluster logs in the time window, and a Cloud Monitoring PromQL query on the node status condition metric, filtered to condition="Ready" and status="Unknown".
If the Ready condition is Unknown, the agent moves to serial port output. Its findings are reported back to you with links to the matching documentation sections, so you can decide what to do next.
(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"Safety notes: scripts, network and read-only rule
This skill does not run scripts. It does use the network, because the agent calls Google Cloud through gcloud and related APIs. The skill's own rule is that the agent runs read-only diagnostic commands only. It must never drain, delete or re-create nodes, or run any other command that changes the cluster.
The skill also asks the agent to tag gcloud commands with a usage attribution variable set inline on each command. The tag only identifies the skill and its version. Review commands the agent proposes before you approve them, especially any remediation step, which you should carry out yourself.
- Runs scripts: no
- Uses network: yes (gcloud and Google Cloud APIs)
- Read-only diagnosis by design; remediation is left to you
Tips for getting useful results
Give the incident time as accurately as you can. The default window is 30 minutes either side of it, and a wrong time will make the logs harder to interpret.
If the node is healthy, expect the agent to conclude that an unresponsive-node timeout is ruled out and to suggest workload checks such as OOM investigation instead. That is a valid outcome.
Keep auto-repair status in mind. The skill points you to the auto-repair documentation for Standard node pools and TPU slice nodes, which is useful when you are deciding how to recover a node.
- Use the exact node pool and node name from kubectl or the console
- Check the time zone: use UTC
- Ask the agent to explain each finding and link its source
What people use it for
- Explaining why a multi-host TPU training job is stuck because one node in a JobSet is NotReady
- Checking whether node auto-repair is enabled on a Standard GPU node pool
- Confirming a kernel panic from serial port output on a node that has been replaced or is stopped
- Ruling out a node timeout so you can focus on an application-level OOM or crash
Frequently asked questions
What is the gke-ai-troubleshooting-node-unresponsive-timeout skill?
It is an agent skill for diagnosing GKE TPU or GPU nodes whose Ready condition is Unknown with NodeStatusUnknown. It guides the agent through read-only checks of node status, Cloud Logging, Cloud Monitoring and serial port output, then points you to likely causes.
How do I install this agent skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder. In the Claude apps you can upload it under Settings → Capabilities → Skills.
What do I need before using it?
You need the Google Cloud SDK (gcloud) and kubectl, an active billing account linked to your project, and the Kubernetes Engine Viewer, Compute Viewer, Logs Viewer and Monitoring Viewer roles. The container, compute, logging and monitoring APIs must be enabled.
Is the skill free and what is its licence?
The skill is listed under the Apache-2.0 licence. The source does not state a price, and Google Cloud usage is subject to your own billing account.
Will the skill fix or restart my nodes?
No. The skill is designed to only run read-only diagnostic commands. It forbids draining, deleting or re-creating nodes, and any fix is handed back to you to apply yourself.
Does this skill work with other AI agents or only Claude?
The source describes installation paths for Claude Code and the Claude apps only. Check the skill's documentation before assuming it works with other agents.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you run multi-host TPU or GPU workloads on GKE and need a structured, read-only way to confirm whether a node has timed out and why. Pick something else if your problem is an application bug in a healthy pod, a routine GKE upgrade, or if you want an agent that repairs or drains nodes for you, since this skill is built not to do that.
Download the skill