Skip to content
Get packs

Guide · Agent skill

How to use the Paperclip Evals Navigator agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The paperclip-evals skill is an instruction file that teaches an AI agent how to work with the evaluation system in the Paperclip repository. It helps the agent pick the right evaluation family for a request, run credential-free checks first, and report results with their source revisions, cost and failure causes. This guide explains what agent skills are, how to install this one, how to trigger it, and what it needs before it is useful.

Paperclip Evals Navigatorpaperclipai/paperclip · MIT

Quick start

  1. 1Get the Paperclip repository. Clone or open a local checkout of the paperclipai/paperclip repository and confirm that pnpm and git are available.
  2. 2Install the skill folder. Unzip the downloaded ZIP and place the paperclip-evals folder in ~/.claude/skills/ or in the project's .claude/skills/ folder, or upload it under Settings, Capabilities, Skills in the Claude apps.
  3. 3Open the repository as your working directory. Run your agent from inside the Paperclip checkout so it can locate the repository with git rev-parse --show-toplevel.
  4. 4Ask a natural question. Describe the evaluation task in plain language, such as choosing a family or summarising a run.
  5. 5Run the credential-free checks. Ask the agent to run pnpm test:e2e:runner:typecheck, pnpm test:e2e:runner:unit and pnpm test:e2e:runner -- --list before anything live.
  6. 6Review the report. Check that source revisions, cost coverage, selected cells and failure classification are included in the agent's summary.

What is an agent skill and what does this one do?

An agent skill is a folder containing a SKILL.md file with instructions that an AI agent reads when a task matches the skill's description. This skill is a set of working rules and pointers to other project documents. It is not a tool that runs evaluations by itself.

When the skill applies, the agent is directed to read doc/evals.md in the Paperclip repository, decide between the two evaluation families, locate the correct checkout, run safe checks, and report results with their provenance attached.

  • Chooses between Runner Evals (runner and provider protocol behavior against the mock control plane) and Product E2E Evals (browser, server, database, runner and provider workflows).
  • Finds the Paperclip checkout explicitly instead of guessing relative paths.
  • Keeps source revisions, fingerprints, model or profile, selected cells, retries, cost coverage and grader version attached to every interpretation.
  • Classifies failures as product, model/provider behavior, grading/evidence, or infrastructure.

Install the paperclip-evals skill

Download the ZIP for the skill from our page and unzip it. Place the resulting paperclip-evals folder in one of two locations. For a personal install, use ~/.claude/skills/. For a single project, use the .claude/skills/ folder inside that project.

You can also upload the skill in the Claude apps. Open Settings, go to Capabilities, then Skills, and upload the skill there.

  • Personal: ~/.claude/skills/paperclip-evals/
  • Project: .claude/skills/paperclip-evals/ inside your project

Meet the requirements before you start

The skill assumes you already have the Paperclip repository on disk. It needs pnpm and git available in the environment. Read-only inspection and credential-free validation need nothing else.

Live runs need more: provider credentials, a configured immutable Daytona image, and the private sibling paperclip-evals checkout or a PAPERCLIP_EVALS_ROOT path pointing to it. Without those, the skill will not launch a live run.

  • A local checkout of the paperclipai/paperclip repository
  • pnpm and git available in your environment
  • For live runs only: provider credentials, a configured immutable Daytona image, and the private paperclip-evals checkout or PAPERCLIP_EVALS_ROOT

Trigger the skill with natural requests

You do not need a special command. Describe the task in plain language while you are working in the Paperclip repository, and the agent should load the skill when the request concerns evaluation selection, interpretation, evidence, history or a live run.

Name the kind of work you want so the agent can route it. Mention whether you want a check, an interpretation, or a live run, since the skill treats live runs differently.

  • "Which eval family should I use for a change to the runner's provider protocol?"
  • "Run the typecheck and unit checks for the Product E2E runner tests."
  • "List the Product E2E runner cells without running them."
  • "Summarise this eval run with its source revision, cost coverage and failure classification."
  • "Does this change belong in a Runner Eval or a Product E2E eval?"
pnpm test:e2e:runner:typecheck pnpm test:e2e:runner:unit pnpm test:e2e:runner -- --list

Start with safe, credential-free checks

The skill tells the agent to begin with read-only catalog inspection and credential-free validation. For Product E2E work these are the typecheck, unit and list commands shown in the code block. A single explicit cell is run only when you have authorized a live or paid run and the credentials and immutable Daytona image are configured.

The skill also says never to treat a partial selector as evidence of full coverage. If an agent reports results from a subset of cells, check whether it says so.

  • Read doc/evals.md first; it defines the two families and their boundaries.
  • Run the listed checks before any live run.
  • Do not accept a partial run as full coverage.

Safety notes: scripts, network and credentials

The skill's own files do not run scripts and do not use the network. It is a set of written instructions. Any commands the agent runs come from the Paperclip repository and its documented workflows, so review them as you would any project command.

The skill forbids launching paid or live provider runs without explicit authorization and configured credentials. It also says not to expose raw trusted artifacts, credentials, secrets, private data, provider session IDs or hidden reasoning in public projections.

  • Scripts run: no. Network use: no.
  • Live provider runs require your explicit approval.
  • Public outputs must not include credentials, secrets, private data or provider session IDs.
  • Refreshing results from retained evidence makes zero provider calls.

What people use it for

  • Deciding whether a change to runner provider behaviour should be tested as a Runner Eval or a Product E2E eval.
  • Listing Product E2E runner cells and validating the test setup without spending credits.
  • Summarising a live eval run for a reviewer, with its provenance and failure classification attached.
  • Interpreting a retained eval result, then refreshing its presentation from evidence without new provider calls.

Frequently asked questions

How do I install the paperclip-evals skill?

Download the ZIP from our page and unzip it. Put the paperclip-evals folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder, or upload it in the Claude apps under Settings, Capabilities, Skills.

What does the paperclip-evals skill require?

It needs a local checkout of the paperclipai/paperclip repository, with pnpm and git available. Live runs also need provider credentials, a configured immutable Daytona image, and the private paperclip-evals checkout or PAPERCLIP_EVALS_ROOT.

Is paperclip-evals free to use?

The skill is published under the MIT licence. Live evaluation runs may involve paid provider usage, which the skill forbids without your explicit authorization.

Does the skill run evaluations by itself?

No. It is a set of working rules and pointers to other project documents. The agent runs commands from the Paperclip repository only in accordance with the skill's instructions.

Can I use this skill outside the Paperclip project?

It is aimed at the Paperclip project. It points to repository-specific files and a private sibling checkout that a general user will not have, so it is not useful for unrelated projects.

What should I do if the agent cannot find the repository?

The skill tells the agent to locate the checkout with git rev-parse --show-toplevel from the current directory, or to select the checkout containing package.json, tests/runner-e2e and packages/paperclip-runner. Run the agent from inside the repository, and set PAPERCLIP_EVALS_ROOT if a Runner Eval needs the private definitions.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you maintain the Paperclip runner or its end-to-end tests and want an agent to interpret eval results consistently, with provenance and failure causes attached. Skip it if you are not working in the Paperclip repository or do not have its private sibling checkout, since it will not help you there.

Download the skill