Guide · Agent skill
How to use the Arize Evaluator Skill agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The Arize Evaluator Skill is an agent skill that teaches an AI agent how to set up automated quality checks on LLM applications hosted on the Arize observability platform. This guide is for engineers and ML teams who already run an application on Arize and want to score its outputs for things like hallucination or correctness. By the end you will know what the skill does, how to install it, how to ask for evaluation work in plain language, and what it needs to run.
Arize Evaluator Skillgithub/awesome-copilot · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the arize-evaluator ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
- 2Install the ax CLI. Install arize-ax-cli with uv, pipx, or pip, and confirm the version is 0.14.0 or higher.
- 3Set up your profile. Create or check your Arize profile with ax profiles show, using your API key from the Arize admin page.
- 4Check your AI integration. Run ax ai-integrations list --space SPACE to confirm an LLM provider integration exists, or create one.
- 5Ask for an evaluation. Describe the evaluator or the evaluation you want in plain language, naming the project or experiment.
- 6Review the results. Check the scores and explanations the agent reports, and treat any failure report as a prompt to fix the issue and retry.
What is an agent skill and what does this one do?
An agent skill is a folder of instructions and supporting files that an AI agent loads when a task matches the skill's description. This one is named arize-evaluator and is published in the github/awesome-copilot repository under the MIT licence.
The skill covers LLM-as-judge evaluation. An evaluator is a judge prompt that asks another AI model to grade outputs, for example whether an answer is hallucinated. A task runs one or more evaluators against real traces, spans, or experiment results, and can keep scoring new data as it arrives.
- Create and update evaluators, including the judge prompt, classification labels, and judge model
- Run evaluations on spans, traces, sessions, or experiments
- Manage tasks, including trigger-run operations and continuous monitoring with a sampling rate
- Map span fields to evaluator template variables such as input and output
Install the skill from the ZIP download
Download the ZIP for the arize-evaluator skill from this page. You can install it for yourself across all projects or for a single project.
For a personal install, unzip the file into ~/.claude/skills/. For a project-only install, unzip it into the .claude/skills/ folder inside your project. If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.
- Personal: ~/.claude/skills/
- Project: .claude/skills/ inside your project folder
- Claude apps: Settings → Capabilities → Skills
Install and configure the ax CLI
The skill drives Arize through the ax command-line tool, so ax must be installed and configured before the skill can do anything useful. The skill's troubleshooting notes say the version must be 0.14.0 or higher. The skill's own guidance is to install with uv, pipx, or pip, and the install command it lists is shown below.
Your Arize credentials are stored as an ax profile. The skill directs the agent to check the profile with ax profiles show and to reference your API key through the ARIZE_API_KEY environment variable rather than pasting the key into a command.
- Install ax using one of the methods the skill lists: uv, pipx, or pip
- Get your API key from https://app.arize.com/admin under API Keys
- Check your profile with ax profiles show
uv tool install arize-ax-cliTrigger the skill with natural requests
You do not need to name the skill. The skill loads when your request mentions terms from its description. Useful trigger phrases include create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, and improve evaluator prompt.
Be specific about the target: the project, the experiment, or the span type, and what quality you want to measure.
- "Create an LLM judge that checks whether answers in my support-bot project are hallucinated."
- "Score the spans in my checkout-agent project using the correctness evaluator."
- "Set up continuous monitoring on my production project with a 10% sampling rate."
- "Map the input and output fields for my faithfulness evaluator in the rag-experiment."
What the agent needs and what it will do
Before the skill can score anything, three things must exist: an Arize account with space access, a configured ax profile, and an AI integration holding your LLM provider key (OpenAI, Anthropic, and others). The skill checks for existing integrations with ax ai-integrations list and asks you for a key only if none exists.
The skill also sets a firm rule: if an evaluation fails, is cancelled, or produces no scores, the agent must report the failure and must not invent scores or run a manual evaluation. It suggests fixing the issue, retrying, running from the Arize UI, checking integrations, or contacting Arize support.
- Arize account with space access
- Configured ax profile with an API key
- AI integration or LLM provider key managed through Arize
- The ax CLI installed and configured
Safety notes: scripts and network access
This skill runs scripts and uses the network. Span content is sent to the judge model, which is a third-party LLM provider you pay for through your own integration. If your traces contain sensitive data, check whether that data may leave your environment before you run an evaluation.
The skill tells the agent not to read .env files or search the filesystem for credentials. Arize credentials should come from ax profiles and LLM provider keys from ax ai-integrations.
- The skill runs scripts and makes network calls to Arize and the judge model provider
- Span content is passed to the judge model
- Credentials are handled through ax profiles and ax ai-integrations, not .env files
What people use it for
- Scoring a support chatbot's answers for hallucination on a live project
- Running a correctness evaluator on an experiment to compare prompt versions
- Continuous monitoring of production traces with a sampled share of new spans
- Evaluating whole agent trajectories at trace granularity
- Tuning an evaluator's prompt after reviewing judge explanations
Frequently asked questions
Is the Arize Evaluator Skill free?
The skill is published under the MIT licence in the github/awesome-copilot repository. The source does not state pricing for Arize or the judge models; the skill drives evaluations that use judge models you pay for through your own integration.
What do I need before I can use this skill?
You need an Arize account with space access, the ax CLI installed and configured, a profile with an API key, and an AI integration or LLM provider key managed through Arize. Without an Arize account or the ax CLI the skill cannot do anything useful.
How do I install the arize-evaluator skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ inside a project. In the Claude apps you can instead upload it under Settings → Capabilities → Skills.
What does the agent do if an evaluation fails?
The skill tells the agent to report the failure clearly and explain the cause. It must not perform a manual evaluation or invent quality scores, and it suggests fixing the issue, retrying, running from the Arize UI, or contacting support.
Does the skill send my data to other companies?
Span content is passed to the judge model, which is run by an LLM provider. The source notes that this data may leave your environment, so review your data sensitivity before running evaluations.
How does this compare to writing my own evaluation scripts?
The skill only drives Arize's own evaluation system and judge models. If you want a general-purpose evaluation method that is not tied to Arize, this skill is not the right tool.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you already run LLM applications on Arize and want an agent to set up hallucination or correctness judges, map span fields, and monitor production traces. Pick something else if you have no Arize account, want a platform-independent evaluation method, or cannot send span content to a third-party judge model.
Download the skill