How to use the GKE TPU vBAR OOM Troubleshooter agent skill

What the skill makes the agent do

The skill covers one specific problem: vbar_control_agent segfaults and out-of-memory (OOM) errors, along with tpu-device-plugin metrics checksum errors and TPU device initialization failures, on TPU v6e nodes in GKE. The source attributes these to race conditions during TPU device resets or high-frequency metrics polling.

Once triggered, the agent follows a fixed sequence. It first establishes the project, cluster, node, workload and namespace, and computes a time window of 30 minutes on either side of the incident timestamp. It then searches serial console logs for OOM messages from vbar_control_agent, searches tpu-device-plugin logs for metrics checksum errors, and checks whether custom TPU metrics collectors are deployed. Finally it recommends one of two resolutions.

• Step 1: look for 'Memory cgroup out of memory' messages for vbar_control_agent in serial console logs.
• Step 2: look for tpu-device-plugin 'metrics fetch failed' errors with 'checksum didn't match' messages.
• Step 3: check whether custom code using libtpu.sdk.tpumonitoring frequently queries GetHostMetrics.
• Resolution 1 (high risk): temporarily disable the custom metrics collector.
• Resolution 2 (low risk): wait for a future GKE version with the vbar_control_agent resiliency update and upgrade then.

What you need before you start

The skill lists its requirements in its own prerequisites. You need Cloud Logging enabled for the project and access to the project and cluster through gcloud or an equivalent tool. The skill's metadata also lists kubectl access to the cluster for the pod listing step, and read access to its reference files.

If you only want the filters and do not have live access, the skill can still help. For informational questions, it outputs the time window, log names and filter templates without running them.

• gcloud CLI with access to the target GCP project and cluster (for live queries)
• kubectl access to the cluster (for the pod listing step)
• Cloud Logging enabled for the project
• Read access to references/attribution.md and references/failure_signatures.md

How to install the skill

Download the skill ZIP from this page and unzip it. For a personal install, place the resulting folder in ~/.claude/skills/. To scope it to one project, place it in that project's .claude/skills/ folder instead.

If you use the Claude apps, you can instead upload the skill under Settings, then Capabilities, then Skills. The folder must contain SKILL.md, which is the file that defines the skill.

• Personal install: ~/.claude/skills/
• Project install: .claude/skills/ inside your project
• Claude apps: Settings → Capabilities → Skills, then upload

How to trigger the skill with a natural request

You do not need a special command. Describe the symptom in plain language and include the incident time if you have one. The skill's description names the signals it looks for, so using those terms helps the agent match the request.

Use a request that describes a failure to get the diagnostic workflow. Use a question about how to check to get only the time window, log names and filter templates, with no live queries.

• 'vbar_control_agent keeps crashing on our TPU v6e nodes in cluster tpu-cluster-prod. The incident was at 2026-04-14T20:00:00Z. Walk me through the diagnosis.'
• 'We see Memory cgroup out of memory messages for vbar_control_agent in serial console logs. Check the logs around that time.'
• 'How do I check whether our tpu-device-plugin is logging checksum errors? Just give me the filter, don't run it.'
• 'Do we have custom code using libtpu.sdk.tpumonitoring that might conflict with the TPU agent?'

Read the Cloud Logging filters the skill uses

The skill's serial console filter searches the vbar_control_agent OOM message in the node's serial port output. The container filter searches tpu-device-plugin errors for metrics fetch failures with a checksum mismatch. Both use the time window the agent computed from your incident timestamp.

The kubectl command lists the image of every container in every namespace, which helps you spot custom monitoring workloads. Because the skill's Step 3 uses it to find collectors, check the output yourself before you act on it.

Understand the two resolutions and their risk

The skill labels its resolutions by risk. Resolution 1 is high risk: it recommends temporarily disabling the custom metrics collector, which prevents reads from vBAR during device resets. The source notes this can reduce visibility into the cluster, so treat it as a change that needs review.

Resolution 2 is low risk: it advises waiting for a future GKE version that makes the agent resilient to memory corruption and able to handle reads from unbound vBARs. The source does not give a release date or version, so the skill cannot tell you when that fix will arrive.

Safety notes: scripts and network use

The skill's metadata says it runs scripts and uses the network. It includes scripts/validate_queries.sh, but the material provided with this page does not describe what that script does, so read it before you run it. The skill also instructs the agent to run gcloud and REST calls against live GCP resources when the request is diagnostic.

The attribution rules matter too. The skill asks that gcloud commands run directly in a terminal carry the CLOUDSDK_METRICS_REQUEST_ATTRIBUTION prefix set inline, not via gcloud config set, so the label does not persist. The tag contains only the skill name and version, with no user data. Commands in generated scripts, Terraform or user code are not supposed to carry it.

• Review scripts/validate_queries.sh before running any part of the skill.
• Run the skill against a non-production project first where you can.
• Do not disable monitoring without review; the skill itself flags this as high risk.

Tips for getting useful results

Give the agent an accurate incident timestamp. The window is 30 minutes before and after it, so a wrong time produces misleading results. Provide cluster and node names as well, because the filters depend on them.

If you already know the filters you need, ask for them without execution. This is the fastest route for a platform team that wants to paste queries into its own tooling. Keep the references files in place, since the skill points to them for example log patterns.

• Provide {issue_time}, {cluster_name} and {node_name} when you can.
• Ask for filters only if you do not want live queries.
• Compare findings with references/failure_signatures.md, which shows an anonymized serial console OOM example and a tpu-device-plugin deserialize failure example.