Add a Runner Eval
Author and validate Paperclip Runner protocol evaluation cases, rosters, and assertions
- Repository
- paperclipai/paperclip
- Stars
- 110,078
- Licence
- MIT
- Files
- 1 file
What it does
This skill guides an AI agent through adding or extending an automated evaluation for the Paperclip Runner, an AI agent runtime. It tells the agent where the eval files live, which existing cases to copy as a model, and which provider-free validation commands to run before anything costs money. It also sets rules: keep one bounded case with a deterministic check, do not grade hidden reasoning or trust a provider's final message as proof of success, and keep private data out of published reports.
Great for
- Paperclip maintainers adding a new Runner protocol test case or roster
- Engineers extending grader assertions and report fixtures in the paperclip-evals repository
- Reviewers who need a consistent, provenance-tracked process for live eval runs
Skip it if
- Browser, server, database, or Daytona end-to-end coverage, which the skill says belongs to a different skill (add-product-e2e-eval)
- Anyone without a local Paperclip checkout and sibling paperclip-evals repository, since the workflow assumes both exist
- Running live provider evals without authorization, because live runs incur model/provider cost and the skill requires pinned commits and a cost record
Try asking
- “Add a new Runner eval case that checks get-task-context behaviour and wire it into the live-mini roster”
- “Extend the grader with a new assertion and calibrate it with positive and negative fixtures”
- “Validate my new case and roster without calling any provider”
- “Update the eval documentation after changing the campaign command contract”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
add-runner-eval.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip add-runner-eval.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip add-runner-eval.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Start with the provider-free validate commands and only request a live run once validation passes and you have approved the cost.
- Copy the nearest existing case, roster, and grader test as a template so new work matches the established conventions.
- When a live run is authorized, pin the exact Paperclip commit and 40-character paperclip-evals commit, and choose the smallest useful roster.
What's inside
- SKILL.md
Redistributed unmodified from paperclipai/paperclip under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.