Skip to content
Get packs

Guide · Agent skill

How to use the Arize Experiment Runner agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The Arize Experiment Runner skill (skill name arize-experiment) gives an AI agent step-by-step instructions for working with Arize experiments through the ax command-line tool. It is written for ML engineers and teams who evaluate or benchmark model versions on Arize and want the agent to drive the ax CLI. After this guide you will know what the skill does, how to install it, how to ask for things in plain language, and what it needs to run.

Arize Experiment Runnergithub/awesome-copilot · MIT

Quick start

  1. 1Download the skill ZIP. Download the ZIP for skills/arize-experiment from the awesome-copilot page.
  2. 2Unzip into a skills folder. Unzip into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Install the ax CLI. Run uv tool install arize-ax-cli, or use pipx or pip. Confirm with ax --version showing 0.14.0 or higher.
  4. 4Set up an Arize profile. Create a profile with your API key from https://app.arize.com/admin under API Keys, and check it with ax profiles show.
  5. 5Ask for an experiment task. Describe what you want in plain language, naming the dataset, the experiment and the space if needed.
  6. 6Review the export. Check the JSON file the agent exports, including outputs, evaluations and metadata, before drawing conclusions.

What the Arize Experiment skill makes the agent do

An agent skill is a folder of instructions (a SKILL.md file plus reference notes) that an agent loads when a request matches the skill's description. This one covers the full experiment workflow on Arize: listing and inspecting experiments, exporting every run to a file, creating experiments from run data, and helping compare how models perform on the same dataset.

The skill defines a few core terms the agent uses. An experiment is a named evaluation run against a specific dataset version, with one run per example. A run holds the model output for one example, plus optional evaluations (named metrics such as correctness, each with a label, score or explanation) and optional metadata.

  • List experiments, optionally filtered by dataset
  • Get metadata for one experiment, including its linked dataset and version
  • Export all runs with their outputs and evaluations to a JSON file
  • Create a new experiment from a CSV, JSON, JSONL or Parquet file of runs

Install the skill in Claude

You can install the skill for your user account, for a single project, or by uploading it in the Claude apps. Download the ZIP for skills/arize-experiment from the awesome-copilot page, then choose one of these routes.

For a personal install, unzip the file into ~/.claude/skills/. For a project-only install, unzip it into the project's .claude/skills/ folder. In the Claude apps, upload the skill under Settings → Capabilities → Skills.

  • Personal: unzip into ~/.claude/skills/
  • Project: unzip into .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload

Install and set up the ax CLI

The skill cannot work without the ax CLI. The troubleshooting reference says the version must be 0.14.0 or higher, and that the package is named arize-ax-cli. The skill suggests installing it with uv, pipx or pip.

You also need a configured Arize profile that holds an API key. The agent is instructed not to check profiles up front; it runs commands and only consults the profile notes when authentication fails. You can check your profile yourself with the command below. If you do not have an API key, the skill points you to https://app.arize.com/admin under API Keys.

  • Install the CLI: uv tool install arize-ax-cli (or pipx install arize-ax-cli, or pip install arize-ax-cli)
  • Confirm the version with ax --version (needs 0.14.0 or higher)
  • Inspect your profile with ax profiles show
ax --version ax profiles show

Trigger the skill with natural requests

You do not need to type a command. The skill loads when your request uses language from its description. Phrases that match include create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models and measure accuracy.

Name the dataset and, if you have several workspaces, the space. The skill accepts a space name or ID and suggests ax spaces list if you are unsure.

  • "Compare gpt-4o and claude on my support-qa dataset and tell me which scores higher on correctness."
  • "Export the results of experiment baseline-v2 on dataset support-qa to a JSON file."
  • "List my recent experiments for the support-qa dataset."
  • "Benchmark our new model against last week's run and summarize the differences."

Export and compare experiment runs

Export downloads all runs to a file in a folder named after the experiment and run time, such as experiment_abc123_20260305_141500/runs.json. The default REST export works over standard HTTPS. For experiments with more than 500 runs, use --all, which uses Arrow Flight over flight.arize.com:443. The skill tells the agent to re-run with --all if a REST export returns exactly 500 runs, because the result is likely truncated.

Once you have the JSON, the agent can compare runs by reading the output, scores and metadata. Each run object has an id, example_id, output, evaluations and metadata such as model name and latency.

  • Default export writes to the current directory; use --output-dir to choose another folder
  • Use --stdout to print JSON instead of writing a file, for example to pipe into jq
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --all ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | jq '.[0]'

Create an experiment from a run file

To upload results you already have, create an experiment with a name, a dataset and a data file. The file can be CSV, JSON, JSONL or Parquet. You can also pipe data in with --file - so no temporary file is needed.

The skill is strict on one point: the agent must call the real model API for every dataset example. It must never fabricate, simulate or hardcode model outputs, latencies or scores. If it cannot call the API because of a missing SDK, missing credentials or a network error, it stops and tells you what is needed.

  • Pass --name, --dataset and --file (all required)
  • Use --space when the dataset is named rather than given by ID
ax experiments create --name "gpt-4o-baseline" --dataset DATASET_NAME --space SPACE --file runs.json ax experiments create --name "claude-test" --dataset DATASET_NAME --space SPACE --file runs.csv

Safety notes: scripts, network and credentials

The skill does not run scripts. It does use the network, because every ax command calls Arize endpoints, and the --all export needs access to flight.arize.com:443. Some corporate networks block that port, and the REST export is the fallback.

The skill also sets rules about credentials. The agent must not read .env files or search the filesystem for keys. Arize credentials come from ax profiles, and LLM provider keys come from ax ai-integrations. API keys are referenced through the ARIZE_API_KEY environment variable, not pasted as flags. The skill does not hold credentials itself.

  • No scripts are bundled to run
  • Network access to Arize endpoints is required
  • LLM provider keys are optional and managed with ax ai-integrations when running experiments

Tips for getting reliable results

Use dataset and space names where you can; the skill prefers names over IDs for readability. Keep the experiment name specific, such as the model and dataset version, so you can find it later with ax experiments list.

When a command fails, read the error before retrying. The troubleshooting notes cover command not found, a version that is too old, and 401 Unauthorized, which usually means the API key is wrong or expired.

  • Use ax experiments list --limit 100 with --cursor to page through many experiments
  • Use -o json or -o csv when you want machine-readable output
  • Ask the agent to summarize a run file rather than pasting raw JSON into chat

What people use it for

  • Comparing two or more model versions on the same dataset to see which scores higher on correctness or relevance
  • Exporting every run of a past experiment to JSON for a separate analysis or report
  • Listing recent experiments for a dataset to find a baseline to compare against
  • Uploading run results produced by your own evaluation code as a new tracked experiment
  • Checking whether a model update changed accuracy before a release

Frequently asked questions

How do I install the Arize Experiment agent skill?

Download the ZIP for skills/arize-experiment from the awesome-copilot page and unzip it into ~/.claude/skills/ for personal use or the project's .claude/skills/ folder. In the Claude apps, you can instead upload it under Settings → Capabilities → Skills.

What do I need before the skill works?

You need the ax CLI (version 0.14.0 or higher, package arize-ax-cli), a configured Arize profile with an API key, and network access to Arize endpoints. LLM provider keys are optional and are managed through ax ai-integrations when you run experiments.

Is the Arize Experiment skill free?

The skill is published under the MIT licence. The source does not state prices for Arize or the ax CLI, so check Arize's own terms for account costs.

Can the skill run my model comparisons without an Arize account?

No. The skill works through the ax CLI and Arize experiments, so it needs an Arize account and API key. For quick one-off comparisons that do not need a tracked evaluation platform, a simpler script may be enough.

Why does my export stop at 500 runs?

The default REST export is limited to 500 runs per page. If you get exactly 500, the skill tells the agent to re-run with --all, which uses Arrow Flight on flight.arize.com:443 for the full set. Some corporate networks block that port.

What should I do if I get 401 Unauthorized?

Run ax profiles show to check the profile. The skill says a 401 usually means the API key is wrong or expired, so update the profile using the ARIZE_API_KEY environment variable and get a key from https://app.arize.com/admin under API Keys if you do not have one.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are an ML engineer or team already on Arize who wants an agent to list, export and compare experiments through the ax CLI. Skip it if you have no Arize account or ax CLI, or if you only need a quick comparison that does not need a tracked evaluation platform, because the skill cannot work without them and does not supply credentials or model calls on its own.

Download the skill