How to use the Agent Evaluation Reporting agent skill

What the skill makes the agent do

The skill turns raw evaluation runs into a report that does not hide failures or overstate capability. It requires every run to be classified exactly once, so autonomous successes, assisted successes, failures, timeouts and invalid runs are counted separately.

Every percentage is shown with its count and denominator, so a reader can reproduce it. The skill also asks the agent to check that the counts add up and to say when a comparison between two agent setups is not fair.

• Autonomous success: the agent satisfied the evaluator without human intervention.
• Assisted success: the task succeeded only after a human intervened.
• Failure: the run reached a terminal, evaluable failure.
• Timeout: the run used up its declared time or step budget.
• Invalid: the agent never got a valid evaluation because the harness, environment or input failed.

Install the agent-evaluation-reporting skill

Download the ZIP file from this page and unzip it. The package contains a single file, SKILL.md. Place the unzipped skill folder in one of these locations:

Option one is your personal skills folder, ~/.claude/skills/, which makes the skill available across your projects. Option two is a project-level .claude/skills/ folder inside the project you are working on, which limits it to that project.

If you use the Claude apps instead, upload the skill under Settings → Capabilities → Skills.

• Personal: ~/.claude/skills/
• Project only: .claude/skills/ inside your project folder
• Claude apps: Settings → Capabilities → Skills

How to trigger the skill with natural requests

You do not need a special command. The skill is meant for requests about summarizing agent evaluations, so describe what you have and what you want. The skill applies when your results include autonomous, assisted, failed, timed-out or invalid outcomes that must stay distinct and comparable.

Give the assistant the raw run data and the context you have: the task set, the agent or prompt version, the timeout and retry policy, and any human-intervention rules. Missing context is not invented by the skill, so the more you supply, the more complete the report.

• Summarize these 120 evaluation runs and keep autonomous and assisted successes separate.
• Compare the new agent version with the old one and tell me whether the comparison is fair.
• Write a pilot readiness report from these logs, with the timeouts and invalid runs still counted.
• Give me the eventual task completion rate and show the denominator for each rate.

Read the report: outcomes, denominators and retries

The skill builds an outcome ledger in which each scheduled attempt has exactly one outcome. It also builds a unique-task rollup, which keeps the first-attempt outcome for each task and one eventual outcome after the retry policy finishes. Retries count as attempts, but a task counts only once in task-level completion.

Each rate is labelled with its denominator. Attempt-level rates are never called workflow completion. If retry records or the retry policy are missing, the report does not state eventual task completion. If a denominator is zero, the rate is reported as unavailable rather than calculated.

Latency is reported by population: autonomous-completion time, assisted end-to-end time and failure time-to-terminal are kept separate, and a success-only median is not presented as an overall median.

• Attempt-level: autonomous and assisted success divided by evaluable attempts.
• Attempt non-completion: failures plus timeouts divided by evaluable attempts.
• Invalid-attempt rate: invalid runs divided by all attempts, including retries.
• Task-level: first-attempt completion, eventual task completion and operational task delivery, each over its own task count.

Worked example

Take 120 unique tasks with one attempt each. Suppose 12 runs were infrastructure-invalid, 48 succeeded autonomously, 24 succeeded with assistance, 20 failed and 16 timed out. The 108 evaluable attempts are the base for the attempt-level rates, and the invalid runs stay in the table as 10 percent of attempts.

Overall latency is marked unavailable because it cannot be rebuilt from subgroup numbers. Readiness is inconclusive until decision gates are declared.

Requirements, limits and safety notes

The skill lists no required tools or software, runs no scripts and makes no network calls, according to its listing. Its output is only as good as the run records you give it. It does not validate the evaluator and does not recreate missing logs, so a report built on incomplete data can look more precise than it is.

Redact credentials, private prompts, personal data and sensitive tool output before you share a report, while keeping stable references to the evidence. Treat critical safety violations as separate release gates instead of averaging them into a quality score. The skill also notes that statistical significance does not establish production safety, user value or acceptable cost, so do not decide production safety from a success rate alone.

• Requires: nothing listed
• Runs scripts: no
• Uses network: no

Tips for honest reports

Declare your pass and fail thresholds before you read the results. Without thresholds, minimum success, maximum timeout rate or latency bounds, the skill reports readiness as not determined and lists the missing gates.

If you want to claim one agent setup caused an improvement, re-run both candidates under the same frozen configuration first. If the conditions differ, the skill reports only a directional observation.

• Give the skill the full run log, including timeouts, invalid runs and retries.
• Name the configuration of each run so the comparison can be checked.
• Ask for failure categories and example evidence next to the headline numbers.