Skip to content
Get packs

Guide · Agent skill

How to use the Skill Autobench Eval Author agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

Skill Autobench is an agent skill that writes a test set, called an eval, for an existing agent skill. Instead of guessing what the skill should do from its written spec, it looks through past conversations and session transcripts for real times the skill was used and where the user had to correct the result. This guide explains what the skill does, how to install it, how to ask for it in plain language, and what it needs before it will run.

Skill Autobench Eval Authorgarrytan/gbrain · MIT

Quick start

  1. 1Download and unzip. Download the skill-autobench ZIP from this page and unzip it to get the skill folder containing SKILL.md.
  2. 2Place the skill folder. Move it into ~/.claude/skills/ for personal use or into .claude/skills/ in your project, or upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Confirm gbrain and the archive. Make sure the gbrain CLI is installed and that conversations/ has been populated by the conversation-archive skill.
  4. 4Ask for an eval in plain language. Name the target skill and ask for an eval built from its usage history, for example by asking to mine how the skill is actually used.
  5. 5Read the staged draft. Open skills/<name>/eval/autobench-<date>.md and check the honesty labels, the grounding warning and the privacy scrub.
  6. 6Approve and run the benchmark. After you approve and merge the contract yourself, convert cases into skillopt-benchmark.jsonl and run gbrain skillopt <name>.

What is an agent skill, and what does skill-autobench do?

An agent skill is a folder containing a SKILL.md file. The file holds instructions that an AI agent loads when a task matches the skill's description or triggers, so the agent follows that procedure instead of improvising.

Skill-autobench works on a different skill, not on itself. It reads the conversation archive and any available session transcripts to find real invocations of the target skill. Each invocation becomes a window that includes the user's request, the invocation itself, and the two turns after it, because that is where corrections usually appear. A user correction after an invocation is treated as the most useful signal, since it records a real failure.

  • Proposes an eval_contract with goal, dimensions and hard_fails, where hard_fails encode observed corrections.
  • Produces 4 to 8 replayable cases, each with an input, an expected behavior and the failure mode it is meant to catch.
  • Labels every dimension and case as HISTORY-IMPLIED (backed by a mined window) or SPEC-DERIVED (inferred from SKILL.md only).
  • Saves the draft as PENDING-HUMAN-APPROVAL. It never rewrites the target skill's SKILL.md.

Install the skill-autobench ZIP in Claude

Download the skill-autobench ZIP from this directory page and unzip it. The skill ships as a single SKILL.md file, so the unzipped folder is what you need to place correctly.

For a personal install, put the skill folder inside ~/.claude/skills/. For a single project, put it inside that project's .claude/skills/ folder. In the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal install: ~/.claude/skills/
  • Project install: .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload

Check that the prerequisites are in place

Skill-autobench depends on the gbrain CLI. It uses the search, query, transcripts, skillopt and eval cross-modal commands. It also needs a populated conversations/ directory, which the conversation-archive skill creates by ingesting your history. If that has not been done, run conversation-archive first, because mining starts with the archive.

Local session transcripts are optional. Claude Code keeps per-project session JSONL files under ~/.claude/projects/, and the skill uses them when they are available. Multi-model judging needs access to several model providers, which the skill file does not name.

  • Required: gbrain CLI with search, query, transcripts, skillopt and eval cross-modal commands
  • Required: a populated conversations/ directory from the conversation-archive skill
  • Optional: local per-harness transcripts, such as ~/.claude/projects/ JSONL files
  • Needed for multi-model judging: access to several model providers
gbrain search "<skill-name>" gbrain query "when did I use <skill-name> and what did I ask for"

How do you trigger the skill with natural requests?

You do not need special syntax. Ask the agent for what you want in plain language, and naming the target skill helps the agent find the right history. The skill's listed triggers include phrases such as the ones below.

Expect the agent to search the brain before reading raw transcript files. If it finds no real invocations, it stops and reports what it checked instead of making examples up.

  • "Write the eval from usage history for my meeting-notes skill"
  • "Mine how this skill is actually used"
  • "Build a benchmark from my corrections"
  • "Synthesize an eval for this skill"
  • "Verify the eval panel" or "did all providers return" to check a multi-model result

Review the staged eval before anything changes

The output is written to skills/<name>/eval/autobench-<date>.md with the status PENDING-HUMAN-APPROVAL. Read it carefully. Check that each HISTORY-IMPLIED label points to a real window, and that the cases match what users actually asked for.

Merging the eval is your decision. The skill's own rules say the agent must not rewrite SKILL.md, but this is a workflow instruction the agent is expected to follow, not a technical lock. Keep the file under review.

  • Look for a GROUNDING WARNING at the top, which means little history was found.
  • Confirm that real names, companies and deals were replaced with placeholder slugs such as alice-example and acme-example.
  • Edit or remove cases that do not reflect the skill's real purpose.

What happens after approval?

Once you approve the eval, you merge its contract into the skill's frontmatter yourself. Approved cases are then converted into lines in skills/<name>/skillopt-benchmark.jsonl and run with gbrain skillopt <name>. Outputs can be judged with gbrain eval cross-modal.

Run the skill again after more usage builds up, and compare the new draft with the earlier staged baseline. The skill also includes a panel integrity check, which asserts that each model in a multi-model judging run returned a non-empty response from a distinct endpoint and did not return byte-identical output to another model.

gbrain eval cross-modal --task "<what the output was meant to achieve>" --output <path>

What people use it for

  • Building a history-grounded test set for a meeting-notes or email-drafting skill from past corrections.
  • Finding out how users actually phrase requests for a skill, so its triggers match real wording.
  • Turning repeated corrections into hard_fail checks that a benchmark can replay.
  • Checking a multi-model judging result to confirm the panel really used different providers and produced different outputs.
  • Tracking how a skill's real usage changes over time by diffing new drafts against an earlier baseline.

Frequently asked questions

Is the skill-autobench agent skill free?

The skill file is published under the MIT licence. This page does not list prices, and the skill's requirements include the gbrain CLI and a populated conversations/ directory, so check those dependencies before you start.

What do I need to install skill-autobench?

You need the skill folder, placed in ~/.claude/skills/ or a project's .claude/skills/, or uploaded under Settings → Capabilities → Skills. It also needs the gbrain CLI and conversation archive data from the conversation-archive skill.

Does skill-autobench change my skill automatically?

No, it stages a draft eval marked PENDING-HUMAN-APPROVAL. The rule that it never rewrites SKILL.md is a workflow instruction the agent must follow, not a technical block, so review the output before merging anything.

What if my skill has never been used?

The skill stops and reports that no history was found, listing what it searched. It does not invent examples. For a skill without usage history, the skill notes that gbrain skillopt with --bootstrap-from-skill is the spec-derived option.

Does skill-autobench send my data over the network?

The listing states that the skill does not use the network, but it runs scripts and reads local conversation and transcript archives. Those can contain private data, so review the staged eval before sharing it, and note that multi-model judging needs access to model providers the skill does not name.

How is this different from writing a benchmark by hand?

A hand-written benchmark usually tests what a spec promised. This skill builds cases from real invocations and user corrections, and labels which items are backed by history and which are only inferred from the spec.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Skill-autobench suits maintainers of agent skills who want tests based on how people really use the skill, and teams running the gbrain self-improving loop. It is a poor fit for skills with no usage history, for anyone expecting automatic updates, or for sensitive environments where the mined output cannot be reviewed before sharing.

Download the skill