Guide · Agent skill
How to use the Statistical Experiment Evaluation agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The Statistical Experiment Evaluation skill is an agent skill that guides an AI agent through designing and recording experiments that test a formal research claim. It is aimed at researchers writing academic papers, students learning to set up baselines and ablations, and teams working in the AutoResearchClaw pipeline at its experiment stages. After this guide you will know what the skill changes about the agent's behaviour, how to install it, how to phrase requests that trigger it, and what it will and will not check for you.
Statistical Experiment Evaluationaiming-lab/AutoResearchClaw · MIT Download the skill Quick start
- 1Download the ZIP. Download the skill ZIP from this page.
- 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ in your project.
- 3Or upload in the Claude apps. In the Claude apps, open Settings, go to Capabilities and upload the skill under Skills.
- 4State your claim. Describe the formal claim or theoretical prediction you want to test.
- 5Ask for the experiment plan. Request an experiment plan covering conditions, baselines, ablations, metrics and failure accounting.
- 6Check the output files. Review the metric rows, run manifest and claim verdicts against the required folder layout.
What an agent skill is and what this one makes the agent do
An agent skill is a folder containing a SKILL.md file with instructions that an AI agent loads when a task matches the skill's description. This skill is meant to be used after the theory and method have been worked out. Its purpose is to make sure experiments test specific claims and theoretical predictions rather than producing numbers with no link to the argument.
When it applies, the agent plans the experiment before running it. The plan covers conditions or data-generating processes, the real data source or synthetic generator, sample sizes, folds, repetitions, seeds or resamples, the proposed method, baselines, ablations, diagnostics, metrics and failure accounting.
- Metrics must map to a formulated claim.
- Comparisons must use the same data conditions across methods.
- Failed runs must be counted.
- Runtime reductions must be recorded.
- Results must be interpreted against theoretical predictions.
Install the skill from the ZIP download or the Claude apps
Option 1: download the ZIP from this page and unzip it into your personal skills folder at ~/.claude/skills/, or into a project's .claude/skills/ folder if you want the skill only for that project. The skill folder should contain SKILL.md.
Option 2: in the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.
- Personal folder: ~/.claude/skills/
- Project folder: .claude/skills/ inside your project
- Source file in the download: SKILL.md
Trigger the skill with natural requests
The skill's trigger keywords include experiment, simulation, evaluation, comparison, baseline, ablation, metrics, diagnostics and statistical evidence. You do not need to name the skill. Ask for the experimental work in plain terms and say which claim it should test. Mentioning the stage of your project helps, since the skill is tied to stages 7 through 14 of the AutoResearchClaw pipeline.
- "Design an experiment that tests whether my proposed estimator beats the standard baseline under heavy-tailed noise."
- "Plan the ablations for claim C2 and list the metrics each one should record."
- "Set up the experiment folder for topic T3 and write the run manifest."
- "Write claim verdicts for C1 and C2 linking Proposition 1 to the simulation results."
Example: the evidence files the agent produces
The skill expects a fixed folder layout under experiments/<TOPIC_ID>/. It includes config.yaml, a src/ folder, results/metrics.json, results/run_manifest.json, results/comparison_summary.md, results/claim_verdicts.json, report/paper.md and a README.md.
Metrics are stored as rows. Each row links a claim, a method, a baseline, a condition, a metric name, a value and a status. Claim verdicts record the verdict together with the theory support, experimental support, comparison, and limitations.
{
"topic_id": "TXX",
"metric_rows": [
{
"claim_id": "C1",
"method": "proposed_method",
"baseline": "standard_method",
"condition": "stress_condition",
"metric": "risk",
"value": 0.12,
"status": "ok"
}
]
}What the skill needs and the safety notes
The skill lists no requirements. It does not run scripts and does not use the network, according to its listing. It only shapes how the agent plans and records experiments. Check your own environment for anything your experiment code itself needs, because the skill does not specify that.
The skill does not verify results. It describes what to record but does not check statistical validity, so you remain responsible for sound sampling, seeds, and interpretation. Its limitations field exists so you can state assumptions that were not tested.
- Runs scripts: no
- Uses network: no
- Licence: MIT
Tips for getting useful output
Give the agent your formulated claims and theory before asking for experiments. The skill assumes a claim exists, so a request with no claim will produce a weaker plan.
Ask for the failure count and runtime comparisons explicitly, since the evidence rules require them to be recorded.
- State the claim ID and the proposition it depends on.
- List the baselines and ablations you already have in mind.
- Ask the agent to write the metric rows and claim verdicts as separate files.
What people use it for
- Planning a simulation study that compares a proposed estimator with a standard baseline across stress conditions.
- Listing ablations and the metrics each should record for a specific claim.
- Writing a run manifest and comparison summary for an experiment folder.
- Drafting claim verdicts that link a proposition to experimental results and state its limitations.
- Teaching a student how to set up fair comparisons using identical data conditions across methods.
Frequently asked questions
How do I install the statistical-experimental-evaluation skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ inside a project. You can also upload it in the Claude apps under Settings, then Capabilities, then Skills.
Does this skill require anything to run?
The listing states no requirements. It does not run scripts and does not use the network, so it only changes how the agent plans and records experiments.
Is the statistical-experimental-evaluation skill free to use?
The skill is published under the MIT licence. The source does not mention pricing, so check the licence text in the repository for the terms.
Does the skill check whether my results are statistically valid?
No. The skill describes what to record, such as metrics, runs and verdicts, but it does not verify statistical validity. You must check sampling, seeds and interpretation yourself.
When should I not use this skill?
Skip it for quick data analysis or exploratory charts, since its artifact structure adds overhead. It is also a poor fit for non-research projects with no formal claim to test.
Why does the agent keep asking for a claim or theory?
The skill is designed to be used after formulation, method proposal and theory, and its rules assume a formulated claim. Give the agent your claim before asking for experiments so each result ties back to it.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you are writing an academic paper or learning experimental design and want each result tied to a specific claim with a disciplined file record. Pick something else for quick exploratory analysis or any project without formal claims, since the artifact overhead will not pay off and the skill will not check your statistics for you.
Download the skill