Skip to content
Get packs

Assay: AI Output Evaluator

Grades AI prompts, models and agents with code checks and calibrated judges, and checks produced artifacts before you see them.

Source Download skill
Repository
danielmiessler/LifeOS
Stars
19,682
Licence
MIT
Files
15 files

What it does

Once installed, the agent can set up and run structured tests on AI output. It writes small code checks and model-graded 'judge' rubrics, runs each case several times, and reports reliability figures such as pass^k, pass@k and confidence intervals. It can also compare prompts or models A/B, review an artifact it just made against what you asked for, and help design multi-question judgment systems. The skill is an eval framework for AI work, not a general test runner.

Great for

  • Developers building LLM features who need regression tests for prompts or agents
  • Teams deciding which model or prompt version performs better on their own tasks
  • People who want an AI to check its own generated documents, images or pages against the original request

Skip it if

  • Anyone expecting a plug-and-play tool: it assumes the LifeOS project layout, and the Bunker CLI is not in the public release, so setup needs manual adaptation
  • Production or blocking decisions based on judge scores without calibration, since the skill itself warns judged results can be provisional and LLM graders can be wrong

Try asking

  • “Create an eval suite that checks my support-reply prompt returns valid JSON and a polite tone, and run it 5 times”
  • “Compare two system prompts A/B on this set of 20 cases and tell me which has the higher pass^k”
  • “Check this generated slide deck against my original brief before I show it to anyone”
Step-by-step guide: how to use Assay: AI Output EvaluatorInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 15 original files, the MIT licence and an attribution note.

    Assay.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip Assay.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip Assay.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Start with deterministic code asserts; reserve judges for checks code cannot answer, since code is free and only code should block a pipeline.
  • Always calibrate a judge against your own labelled examples and keep an adversarial case it must fail.
  • Ask for the requested and completed trial counts alongside any score, so you can see how much evidence supports it.

What's inside

  • Intents/Calibrate.md
  • Intents/Compare.md
  • Intents/CreateJudge.md
  • Intents/CreateQuestions.md
  • Intents/CreateSuite.md
  • Intents/ModelRetro.md
  • Intents/Proof.md
  • Intents/RunAgentic.md
  • Intents/RunEval.md
  • Intents/ViewResults.md
  • Libraries/blog-post.yaml
  • Libraries/code-change.yaml
  • Libraries/image.yaml
  • Libraries/video.yaml
  • SKILL.md

Redistributed unmodified from danielmiessler/LifeOS under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.