How to use the Add a Runner Eval agent skill

What the skill makes the agent do

The skill guides an agent through adding or extending a Runner protocol evaluation definition, roster, assertion, or report fixture. It tells the agent where the eval files live: definitions and authored cases belong in the sibling paperclip-evals/evals/paperclip-runner directory, while Runner integration and publication behavior belong in packages/paperclip-runner.

The skill also sets the rules. The agent should author one bounded case with a deterministic semantic assertion, avoid grading hidden reasoning, and not treat a provider's final message as proof of success. Public reports must not expose credentials, secrets, private references, raw trusted payloads, or hidden reasoning.

• Cases live under cases/, company fixtures under fixtures/, runtime and model settings under configs/, and selections under rosters/.
• A new case file does not join the maintained campaign by itself; campaign membership in campaigns/live-direct-full.json may need updating.
• Browser, server, database, and Daytona end-to-end coverage belongs to a different skill, add-product-e2e-eval.

Install the add-runner-eval skill

Download the ZIP for this skill from the page you are reading. Unzip it so that the skill folder, which contains SKILL.md, ends up in one of the following locations.

For a personal install, put it in ~/.claude/skills/. For a project-only install, put it in the project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

• Personal: ~/.claude/skills/ (available across your projects).
• Project: .claude/skills/ inside the project root (shared with that project).
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP.

Check your environment before you start

The skill assumes two repositories exist on your machine. You need a local Paperclip git checkout that contains packages/paperclip-runner and tests/runner-e2e, and a sibling paperclip-evals repository. The agent finds the Paperclip root from the PAPERCLIP_ROOT variable or from git rev-parse --show-toplevel, and finds paperclip-evals from PAPERCLIP_EVALS_ROOT or a discovered sibling.

You also need Python 3 to run the eval tool scripts, and Git to locate the repository root and pin commits. Live runs need provider credentials and accounts, but the SKILL.md does not specify which ones, so check the Runner docs for that.

• A local Paperclip git checkout with packages/paperclip-runner and tests/runner-e2e.
• A sibling paperclip-evals repository, or PAPERCLIP_EVALS_ROOT pointing to it.
• Python 3 and Git.
• Provider credentials, only for live runs.

Trigger the skill with natural requests

You do not need a special command. Describe the eval change you want in plain language and mention Runner or the eval itself, so the agent matches the request to this skill. Be specific about the case or roster you want to touch.

Good requests name the outcome: the case you want, the assertion you want to calibrate, or the report fixture you need. Vague requests such as 'make the tests better' are less likely to lead to a bounded, reviewable change.

• Add a new Runner protocol eval case that checks get-task-context output.
• Extend the grader assertion for a Runner eval and add positive and negative fixtures.
• Create a small live-mini roster entry for a new Runner case and validate it without provider calls.
• Add a report fixture for a Runner eval that uses only sanitized conversation.

Validate without provider calls first

Run the provider-free checks before any live run. From the paperclip-evals repository root, the skill lists four commands: validate the case and config with eval_program.py, validate a roster with run_live_roster.py, validate the campaign with run_live_campaign.py, and validate the reliability plan with reliability_campaign.py. Run the reliability-plan validator only when that separate plan changes.

Adapt the commands to the case and roster you changed. The example below uses get-task-context as the case and live-mini as the roster.

• Use nearby positive and negative grader cases to confirm the grader fails closed on malformed or missing evidence.
• When a live run is authorized, pin the Paperclip commit and the exact 40-character paperclip-evals commit, select the smallest useful roster, and keep the full provenance and cost record.

Safety notes: scripts and network

The skill runs scripts, so review what it executes before you approve it, especially the validation commands and any live-run commands. The assessment for this item states that the skill runs scripts and does not use the network. Live provider runs are different: they incur model or provider cost and need explicit authorization.

Keep private data out of published output. The skill's rule is that only sanitized fixture conversation and allowlisted tool outcomes may be published. Raw trusted artifacts, credentials, and private references stay out of public reports.

• Never start a live provider run without authorization; it costs money.
• Pin the Paperclip commit and the 40-character paperclip-evals commit before a live run.
• Do not publish raw trusted artifacts, credentials, secrets, or private references.
• Public reports are immutable; use the reviewed projection.

Tips for a clean eval change

Keep each change to one bounded case with one deterministic check. Copy the nearest existing case as your model rather than starting from scratch, and read doc/evals.md and the Runner live-evals doc first. When the contract or command changes, update the authoritative detailed docs and link to them from doc/evals.md instead of duplicating the runbook.

• Declare the expected operation, state effect, provider lane or profile, timeout and retry policy, and any required evidence in the case.
• Do not infer success from a provider's terminal message.
• Preserve existing machine dispositions and grades; labels are annotations, not instructions to rewrite classifiers.