Skip to content
Get packs

Guide · Agent skill

How to use the Add Product E2E Eval agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The add-product-e2e-eval skill is an agent skill, a packaged set of instructions that an AI agent such as Claude loads when you ask for a matching task. It guides the agent through adding or changing a Product E2E eval inside the paperclipai/paperclip repository. This guide is for engineers and contributors who maintain Paperclip's runner end-to-end tests, and it explains how to install the skill, trigger it, and stay within its safety limits.

Add Product E2E Evalpaperclipai/paperclip · MIT

Quick start

  1. 1Download and unzip the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into .claude/skills/ inside your project. In the Claude apps you can upload the ZIP under Settings → Capabilities → Skills instead.
  2. 2Get a Paperclip checkout. Clone or open the paperclipai/paperclip repository locally. Set PAPERCLIP_ROOT if you want to point the agent at a specific checkout.
  3. 3Ask for one bounded change. Describe a single user journey and the outcome it should verify, mentioning Paperclip and Product E2E.
  4. 4Confirm discovery. Run pnpm test:e2e:runner -- --list --suite <suite-name> so the new case appears in the suite.
  5. 5Run credential-free checks. Run the typecheck and unit commands before any live run. These do not need provider or Daytona credentials.
  6. 6Authorize a minimal live run. If you approve it, run only the smallest explicit --id selection. Review cost and credential handling first.
  7. 7Generate and review the report. Run pnpm test:e2e:runner:report and check the packaged attempt evidence and dashboard.

What the skill makes the agent do

Product E2E evals run a real Chromium browser against the Paperclip server, a database, the runner and a model provider, optionally on Daytona cloud sandboxes. Each eval checks whether a user workflow actually succeeded.

The skill tells the agent to define one bounded workflow with a clear user outcome, register it in tests/runner-e2e/catalog.ts and the relevant fixture-registry.ts paths, and follow the existing catalog, fixture, matcher and report patterns. It also tells the agent to run credential-free checks before any paid or live run.

  • Keep the user journey on production browser and API surfaces, with no private runner hooks or direct fixture database writes.
  • Keep credentials and secrets out of catalog data, screenshots, fixture metadata, logs and tracked files.
  • Use Daytona images only as immutable digest references.
  • Protocol-level runner tests against the mock control plane belong to the separate add-runner-eval skill.

Install the add-product-e2e-eval skill

Download the skill ZIP from this page. Then choose one of two install locations: a personal skills folder, or a project-level folder inside a specific repository.

For a personal install, unzip the file into ~/.claude/skills/. For a project install, unzip it into the .claude/skills/ folder of that project. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/
  • Project: .claude/skills/ inside the project folder
  • Claude apps: Settings → Capabilities → Skills

Prepare a local Paperclip checkout

The skill needs a local checkout of the paperclipai/paperclip repository, plus pnpm and git. The agent locates the repository using the PAPERCLIP_ROOT environment variable when it is set, or with git from inside a checkout. The copied skill can live outside the checkout.

Before changing anything, the agent is directed to read doc/evals.md and the authoritative tests/runner-e2e README, FIXTURES, SECURITY and EVERYDAY-WORKFLOWS files for the area it is changing. It must also confirm that the selected root contains tests/runner-e2e and packages/paperclip-runner.

  • Requires a local checkout of paperclipai/paperclip
  • Requires pnpm and git
  • Chromium is needed for live browser runs
  • A model provider and credentials are needed only for live runs
  • Daytona needs an account, credentials and an immutable image digest
git rev-parse --show-toplevel

Trigger the skill with natural requests

You do not need a special command. Ask the agent in plain language for the kind of change you want, and mention Paperclip and the Product E2E area so the skill matches. Name the user journey you want covered, since the skill is built around one bounded workflow at a time.

Good requests name the outcome and the scope. Vague requests tend to produce broad changes, so narrow them to one case.

  • Add a Product E2E eval that checks a user can create a project and see it on the dashboard in Paperclip.
  • Extend the everyday workflow case so it also verifies the uploaded file appears in the evidence report.
  • Add a fixture and matcher for a Paperclip Product E2E case that runs on Daytona with an immutable image digest.
  • Review how the Product E2E report classifies a wrong workflow outcome in the existing suite.

Run the checks the skill expects

The skill asks for discovery and credential-free checks first. Listing confirms that a new case is registered before anything runs. Typecheck and unit tests run without live credentials.

A live run is only for an authorized, smallest explicit selection. The skill describes a full --all campaign as paid, and it excludes the manual everyday-workflows suite and other explicit-only cells, so those must be named exactly.

  • Confirm discovery with pnpm test:e2e:runner -- --list --suite <suite-name>
  • Check the manual suite with pnpm test:e2e:runner -- --list --suite everyday-workflows
  • Run pnpm test:e2e:runner:typecheck and pnpm test:e2e:runner:unit before any live run
  • Generate the Product E2E report with pnpm test:e2e:runner:report
pnpm test:e2e:runner:typecheck pnpm test:e2e:runner:unit pnpm test:e2e:runner -- --list

Safety notes: scripts, network and credentials

This skill runs scripts and uses the network. Its commands run tests that start a browser, a server, a database and a provider connection, and live runs can incur cost. Treat any live run as a decision you authorize, not a default.

The skill tells the agent not to publish raw trusted traces, videos, archives, databases, workspaces, credentials, provider session IDs or hidden reasoning. Results should be verified from the packaged attempt evidence and dashboard, because passing model-authored tests cannot override the independent oracle.

  • Review the commands the agent proposes before approving a live run.
  • Keep provider and Daytona credentials out of tracked files.
  • Startup, transport and timeout failures need evidence before attribution to Paperclip, the provider or infrastructure.
  • Do not claim qualification from a partial or manual selection.

What people use it for

  • Adding a new user-journey case to the everyday workflows area with grading and evidence paths.
  • Extending an existing Product E2E fixture or matcher so it checks more durable state.
  • Adding a Daytona-backed case that uses an immutable image digest.
  • Reviewing whether a wrong workflow outcome is classified as product or model/provider behavior.

Frequently asked questions

Is the add-product-e2e-eval skill free?

The skill is published under the MIT licence. Live runs are a separate matter: the skill describes full campaigns as paid, and provider and Daytona usage can incur cost.

What do I need to install and use this skill?

You need a local checkout of the paperclipai/paperclip repository, pnpm and git. Chromium is needed for live browser runs, and a model provider, plus Daytona credentials and an immutable image digest, only if you run those.

Where do I put the skill files?

Unzip the downloaded ZIP into ~/.claude/skills/ for a personal install, or into .claude/skills/ in a project. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

Can I use this skill for general web testing?

No. The skill is specific to the Paperclip repository and its test layout, including tests/runner-e2e and packages/paperclip-runner. For general web testing, use a tool designed for that purpose.

Which skill should I use for protocol-level runner tests?

Use add-runner-eval. The add-product-e2e-eval skill says runner protocol cases against the mock control plane belong to that separate skill.

Why did my new eval not show up in the list?

Run pnpm test:e2e:runner -- --list --suite <suite-name> to confirm discovery. The everyday-workflows suite and other explicit-only cells are excluded from --all, so name them exactly with --suite or an execution ID.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits engineers and contributors who work on Paperclip's runner end-to-end suite and need a guided, repo-specific path for adding evals with grading and evidence. It is a poor fit for general web testing, for anyone who wants live paid runs without reviewing cost and credentials, or for protocol-level runner tests, which belong to add-runner-eval.

Download the skill