Skip to content
Get packs

Guide · Agent skill

How to use the Assay AI output evaluator agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

Assay is an agent skill for evaluating AI output. Once it is installed, your agent can write small code checks and model-graded judge rubrics, run each test case several times, and report reliability numbers such as pass^k and pass@k. It is aimed at developers building LLM features, teams comparing prompts or models on their own tasks, and anyone who wants an AI to check its own generated work against the original request. After this guide you will know what the skill does, how to install it, how to trigger it, and what it needs before you rely on it.

Assay: AI Output Evaluatordanielmiessler/LifeOS · MIT

Quick start

  1. 1Download the ZIP. Get the Assay ZIP from this page.
  2. 2Unzip into a skills folder. Unzip into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder.
  3. 3Install Bun and set up paths. Install the Bun runtime and make sure the LifeOS paths exist, or adapt the paths manually.
  4. 4Connect a model API. Provide your own model access for judge and agent runs, because provider keys are not included.
  5. 5Describe the eval in plain words. Ask the agent to run an eval, compare models, or proof an output, using natural phrasing.
  6. 6Read the reliability numbers. Check pass^k, SEM and the 95% CI, and note the completed trial count before trusting a score.

What is an agent skill, and what does Assay make the agent do?

An agent skill is a folder of instructions and supporting files that an AI agent loads when a request matches the skill's trigger phrases. Assay turns the agent into an evaluation harness for AI work. An eval gives an AI an input, then applies assertions to the output to measure success.

Each assertion is either deterministic, meaning code that runs for free, or model-graded, meaning a calibrated judge. Cases run multiple trials and report pass^k (every trial passes, the reliability and regression measure) and pass@k (any trial passes). The skill also reports SEM and a 95% confidence interval from the completed trials.

  • Write code checks and LLM-graded rubrics for a prompt, model or agent
  • Run each case several times and report pass^k, pass@k, SEM and 95% CI
  • Compare two configurations A/B on the same cases with a paired confidence interval
  • Check an artifact the agent just produced against what you asked for, before you see it
  • Help design multi-question judgment systems for fuzzy decisions

Install the Assay skill from the ZIP or the Claude apps

Download the Assay ZIP from this page. For a personal install, unzip it into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder. The skill is then available to agents that read those locations.

If you use the Claude apps instead, open Settings, go to Capabilities, then Skills, and upload the ZIP there.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into the project's .claude/skills/
  • Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Check the requirements before you start

Assay assumes a particular environment. Read these requirements before you install, because setup can need manual work.

The bundled commands use the Bun runtime. The skill also expects the LifeOS directory layout, with paths such as LIFEOS/ASSAY, or you must adapt the paths by hand. Judge and agent runs need access to a model API, but the provider keys are not specified in the files shown, so you must supply your own access.

The Bunker CLI is not part of the public release. The skill's own notes say to run code rows with bun test on a path, and suite rows with bun skills/Assay/Tools/EvalRunner.ts.

  • Bun runtime installed (the bundled commands use bun)
  • LifeOS directory layout (LIFEOS/ASSAY paths), or manual path changes
  • Access to a model API for judge and agent runs; provider keys are not included
bun skills/Assay/Tools/EvalRunner.ts

Trigger the skill with natural requests

You do not need a special command. Describe what you want in plain language and the agent routes the request to the right intent. The skill's trigger phrases include words such as eval, judge, benchmark, pass^k, compare prompts or models, A/B test, check the output and design a pipeline.

Each intent has its own file. Proof checks a produced artifact. RunEval runs a suite. CreateSuite builds a new eval. CreateJudge writes a model-graded rubric. Calibrate checks a judge against your labels. Compare runs an A/B test. ModelRetro looks back at how models performed. RunAgentic covers multi-turn and tool-using agents. ViewResults shows scores. CreateQuestions designs judgment question systems.

  • "Run the eval for the summarizer suite"
  • "A/B test prompt v1 against prompt v2 on the support suite"
  • "Compare these two models on our own tasks"
  • "Proof this deck against what I asked for"
  • "Write a judge that checks whether the answer cites its sources"
  • "Calibrate the judge against the labels I gave you"

Use Assay on a produced artifact, a suite or an A/B test

The most direct use is checking an artifact. Ask the agent to proof a document, image, page or email it made. Proof applies the same check tiers to that output and compares it with your original request before you see it.

For a regression suite, ask for a new suite for a feature, then run it on each change. Read the pass^k figure first. For an A/B test, name both configurations and the suite. The skill requires a paired confidence interval over the same cases, and it will not call a winner from two separate scores.

Treat judge scores as provisional until the judge has been calibrated against labels you gave it. The skill warns that judged results can be provisional and that LLM graders can be wrong.

  • Start with code asserts: they are free and reliable
  • Use judges only where code cannot express the property
  • Read pass^k for reliability, not just the average

Safety notes: scripts and network access

Assay runs scripts and uses the network. Its evals call model APIs for judge and agent runs. The skill's deterministic check runs a TypeScript validator with bun on an example question-system file.

Review the files before you run them, and run them on data you are comfortable sending to your model provider. Keep provider keys in your own environment rather than in shared project files.

  • Runs scripts: yes (for example, bun on the question-system validator)
  • Uses network: yes (model API calls for judges and agent runs)
  • Assay is not a general test runner and is not a plug-and-play tool
bun LIFEOS/ASSAY/QuestionSystem.ts validate LIFEOS/ASSAY/fixtures/question-system.example.json

What people use it for

  • Regression-testing a prompt so a change does not break cases that used to pass
  • Deciding which model performs better on your own support or summarization tasks
  • Checking a generated document, image or web page against the original brief before handing it over
  • Designing a judgment question system that replaces a fuzzy decision with measured questions

Frequently asked questions

Is the Assay skill free?

The skill is published under the MIT licence, as stated on this page. Model API calls used by judge and agent runs are billed by your model provider, and the source does not give prices.

What do I need to install Assay?

You need the Bun runtime, the LifeOS directory layout or manual path changes, and access to a model API with your own keys. The Bunker CLI is not in the public release.

How do I install the Assay skill in Claude?

Unzip the ZIP into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. In the Claude apps, go to Settings, Capabilities, Skills and upload the ZIP.

Can Assay work without LifeOS?

The skill assumes the LifeOS layout, so you would need to adapt the paths by hand. The source does not describe a standalone setup, so expect manual work.

What is the difference between pass^k and pass@k?

pass^k counts a case as passing only if every trial passes, which measures reliability and works as a regression gate. pass@k counts a case as passing if any trial passes.

Are the judge scores reliable enough to block releases?

The skill says judged results can be provisional and LLM graders can be wrong. Only deterministic code checks should sit on a blocking path, and judges should be calibrated against your labels first.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Assay suits developers and teams who want repeatable, statistically reported tests for prompts, models and agents, and who can handle Bun and model API setup. If you want a plug-and-play tool or a general test runner, or you need to make production decisions from judge scores without calibration, pick something else.

Download the skill