Skip to content
Get packs

Arize Annotation Manager

Set up label schemas and human review queues on Arize and bulk-apply annotations to spans.

Source Download skill
Repository
github/awesome-copilot
Stars
40,703
Licence
MIT
Files
3 files

What it does

This skill teaches an AI agent how to manage human feedback on Arize, a platform for monitoring AI apps. The agent can create annotation configs (the label definitions, such as correct/incorrect, a 0-10 score, or free-text notes), set up annotation queues that route records to human reviewers, and apply annotations to project spans through the Python SDK. It relies on the ax command-line tool and a configured Arize profile to act on your account.

Great for

  • ML or LLM engineers who run evaluation and labeling workflows on Arize
  • Data teams organising human review of AI outputs
  • Teams that need to script bulk annotation of traced spans

Skip it if

  • Users without an Arize account or the ax CLI, since nothing works without them
  • Anyone wanting to label data outside Arize, because the skill only targets Arize spaces, configs and queues
  • Users who might paste API keys into chat: the skill tells the agent to use ax profiles instead, but credentials still pass through the agent's session

Try asking

  • “Create a categorical annotation config called Correctness with correct and incorrect values”
  • “Set up an annotation queue for reviewers to label a support-bot project, assigned to two emails”
  • “Bulk-apply human annotations to spans in my project using update_annotations”
  • “List the records waiting in my Correctness Review queue”
Step-by-step guide: how to use Arize Annotation ManagerInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    arize-annotation.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip arize-annotation.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip arize-annotation.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Create the annotation config in the space before expecting labels to persist, since queues and labels depend on it
  • Remember that list flags such as --annotation-config-id fully replace existing values on update, so pass every desired value
  • Deleting a config is irreversible, so confirm before using --force

What's inside

  • references/ax-profiles.md
  • references/ax-setup.md
  • SKILL.md

Redistributed unmodified from github/awesome-copilot under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.