How to use the Agent Platform Eval Flywheel agent skill

What is an agent skill, and what does this one do?

An agent skill is a folder containing a SKILL.md file and supporting files that tell an AI agent how to handle a particular kind of task. When the skill is installed and your request matches its description, the agent follows its method instead of improvising.

This skill covers the quality loop for GenAI agents on Google Cloud. The agent builds test scenarios and datasets, chooses scoring metrics, runs LLM-as-judge evaluations, reads the failures, and compares results before and after a fix. It is built around the Agent Platform GenAI Evaluation SDK.

• Prepare data: build an evaluation dataset from test cases, pandas DataFrames, or session traces.
• Choose metrics: select or write scoring metrics and read rubric verdicts.
• Run evaluations: call the SDK's evaluate step, with confirmation before paid compute.
• Analyze failures: cluster problems and map common failure causes to fixes.
• Compare: check results before and after a change to see regressions.

Which helper scripts does the skill include?

The skill ships helper scripts for validating datasets, inspecting and comparing results, parsing agent traces, rendering HTML reports, and evaluating models served on an endpoint or as Model-as-a-Service (MaaS).

• scripts/validate_dataset.py validates a dataset.
• scripts/inspect_results.py inspects results.
• scripts/compare_results.py compares results.
• scripts/parse_adk_traces.py parses agent traces.
• scripts/render_html_report.py renders HTML reports.
• scripts/endpoint_evaluation.py and scripts/maas_evaluation.py evaluate models served on an endpoint or by model ID.

Install the skill from a ZIP file

Download the ZIP for this skill from the page you are reading. Then choose one of two install locations, depending on whether you want the skill for all your projects or one project only.

For a personal install that applies everywhere, unzip the file into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder. If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.

• Personal: unzip into ~/.claude/skills/
• Project only: unzip into .claude/skills/ inside your project
• Claude apps: Settings → Capabilities → Skills, then upload

Set up Python packages and Google Cloud settings

The scripts need Python 3, the vertexai package from google-cloud-aiplatform[evaluation], google-genai, pandas, and requests. The skill asks the agent to check for these first and install only what is missing, rather than creating a new virtual environment.

You also need a Google Cloud project and location, set through the GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION settings, and credentials with access to Agent Platform. Endpoint evaluation uses a gcloud access token.

• Required: Python 3, google-cloud-aiplatform[evaluation] version 1.163.0 or later, google-genai version 1.0.0 or later, pandas, requests
• Required: Google Cloud project and location
• Required: Google Cloud credentials with Agent Platform access

Trigger the skill with a natural request

You do not need special syntax. Describe the task in plain words that mention evaluating, measuring, or comparing an agent or model. The agent will match your request to the skill and follow its steps.

If your request leaves out the project or location, the skill tells the agent to stop and ask for them before preparing an evaluation. It also keeps the location you give it and will not switch to global unless you ask.

• Evaluate my customer support agent on Google Cloud for answer grounding.
• Build an eval dataset from these session traces and check it is valid.
• Compare the results from before and after my prompt change.
• Evaluate the model on my endpoint in us-central1 for my project.

Confirm each paid evaluation run

Evaluation steps that call LLMs or remote services consume compute and incur costs. These include running inference, running evaluate, generating conversation scenarios, and generating loss clusters. The skill requires a dry-run preview and an explicit Yes or No from you before each such run.

Every new evaluation, re-evaluation, metric change, parameter change, or synthetic scenario generation needs its own preview and approval. The agent will not run these steps in the same turn as the confirmation prompt; it waits for your reply. Read-only helper scripts run without confirmation.

• Read-only scripts (inspect, compare, validate, parse traces, render reports) run without confirmation.
• Paid operations need a preview card and your explicit approval each time.
• Approval words such as Yes, Approved, Go ahead, or Proceed let the agent run the previewed step.

Safety notes: scripts and network access

This skill runs scripts and uses the network. Its scripts call Google Cloud services, and the evaluation steps send requests to remote services. Before approving a run, read the preview to see the project, location, model, and scope.

Keep credentials in your own environment and review the scripts if your organisation requires it. The skill is licensed under Apache-2.0.

• It runs scripts and makes network calls.
• Each paid run needs your approval after you read the preview.
• Remote evaluation and synthetic scenario generation incur compute costs.

Tips for a repeatable quality loop

Plan for several passes. The skill's method expects 5 to 10 or more iterations, because stopping early can leave regressions on other metrics unnoticed. Do not lower a metric threshold to make a failing case pass; fix the agent instead.

When a case is flaky, the skill suggests reducing non-determinism with temperature=0 or stricter instructions. If expected outputs keep changing, the agent likely has a behavior problem rather than a dataset problem.

• Always run the evaluation and read the scores, rather than judging from a trace.
• Compare results before and after each change.
• Use the reference files for dataset schema, metric registry, deployment, and failure patterns.