Skip to content
Get packs

Skill Autobench Eval Author

Writes a test suite for an existing agent skill from its real usage history and corrections

Source Download skill
Repository
garrytan/gbrain
Stars
31,447
Licence
MIT
Files
1 file

What it does

Lets an agent look through past conversation records and session transcripts to find real times a given skill was used, and where the user had to correct the result. It turns those moments into a proposed evaluation with a few replayable test cases, labels each item as backed by history or inferred from the spec, and saves the draft as pending for a human to approve. It never edits the skill's own instructions.

Great for

  • Maintainers of agent skills who want tests based on how users actually use the skill
  • Teams running the gbrain self-improving loop who need a history-grounded benchmark

Skip it if

  • Skills with no usage history: the skill stops and reports no history rather than inventing examples, so a different tool (spec-derived bootstrap) is needed
  • Anyone who expects it to automatically update a skill: it only stages a draft for human approval, and the no-rewrite rule is a workflow instruction the agent must follow, not an enforced technical gate
  • Sensitive environments without review: it reads local conversation and transcript archives, which can contain private data, so the output should be checked before sharing

Try asking

  • “Build a benchmark from my corrections for the invoice-summary skill”
  • “Synthesize an eval for this skill from how I actually used it last month”
  • “Mine how this skill is actually used and draft test cases”
  • “Verify the eval panel: did all providers return a response?”
Step-by-step guide: how to use Skill Autobench Eval AuthorInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 1 original files, the MIT licence and an attribution note.

    skill-autobench.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip skill-autobench.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip skill-autobench.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run the conversation-archive skill first so the conversations/ substrate exists; otherwise mining yields nothing.
  • Review the staged autobench file carefully, especially the honesty labels and the GROUNDING WARNING, before merging anything into the skill.
  • Keep the cheap-tier default for mining and synthesis; enable the full multi-model judging pass only when you need it, since it costs more.

What's inside

  • SKILL.md

Redistributed unmodified from garrytan/gbrain under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.