Guide · Agent skill
How to use the A/B Test Setup agent skill (ab-test-setup)
Updated · written with AI from the project's own documentation and checked against our catalogue data
The ab-test-setup skill is a set of written instructions that tells an AI agent how to design an A/B (split) test for a product change. Once installed, the agent works through a fixed procedure: it states one hypothesis, picks a primary metric, estimates the sample size, checks tracking and fixes its stopping rules before any variant is built. This guide covers what the skill does, how to install it, how to trigger it, its requirements and safety notes.
A/B Test Setupsickn33/agentic-awesome-skills · MIT Download the skill Quick start
- 1Download and unzip. Download the skill ZIP from the directory page and unzip it so that the ab-test-setup folder containing SKILL.md is available.
- 2Place the skill folder. Move the folder to ~/.claude/skills/ for all projects, or to .claude/skills/ inside one project. Alternatively, upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
- 3Describe your experiment. Ask the agent in plain language to design an A/B or split test, and include the product question, the change and your rough traffic.
- 4Lock the hypothesis. Review the hypothesis, audience, primary metric, expected direction and MDE the agent presents, and confirm or correct them.
- 5Estimate sample size. Replace the illustrative baseline and variant rates with your own, then run the Python example with statsmodels-free code to get a per-variant sample.
- 6Verify tracking. Work through the five tracking checks on staging, including event firing, variant attribution, de-duplication, randomization and guardrail alerts.
- 7Run and record. Run the test without early stopping or mid-test changes, then write the test record with the decision and learnings.
What is an agent skill and what does this one make the agent do?
An agent skill is a folder containing a SKILL.md file with instructions that an AI agent loads when a task matches the skill's description. This skill is described as: use when designing an A/B or split test, to define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.
Before any variants or metrics are designed, the skill requires the agent to present a final hypothesis that specifies the target audience, primary metric, expected direction of effect and minimum detectable effect (MDE). It also requires an assumptions check covering traffic stability, user independence, metric reliability, randomization quality and external factors such as seasonality, campaigns or releases.
- Checks that a hypothesis has an observation, a single specific change, a directional expectation, a defined audience and measurable success criteria.
- Chooses the simplest valid test type, defaulting to a standard A/B test.
- Defines a single primary metric, secondary metrics for context, and guardrail metrics that must not degrade.
- Requires a sample size estimate and expected duration before implementation.
How do you install the ab-test-setup skill?
Download the ZIP for this skill from the directory page and unzip it. The skill is a folder that contains SKILL.md. Place that folder in one of these locations:
- Personal skills for all your projects: ~/.claude/skills/
- Project-only skills: the .claude/skills/ folder inside your project
- Claude apps: upload the ZIP under Settings → Capabilities → Skills
How do you trigger the skill with a natural request?
You do not need a special command. Describe the experiment in plain language and mention a test, split test or A/B test. The agent matches your request against the skill description and loads the procedure. Give it the product question, the change you want to test and any traffic or analytics details you have, because the skill assumes these exist.
- "Help me design an A/B test for a shorter signup form. We get about 5,000 visitors a week."
- "Plan a split test on our pricing page and tell me what the primary metric and stopping rule should be."
- "Check whether our event tracking is good enough to run the checkout button experiment."
What do you need before using it?
The skill itself is text, but its sample-size example is Python 3 code that references the statsmodels library. You also need access to an analytics source and a rough estimate of traffic volume. The skill assumes you have a clear user problem; without one, it will ask you to define it.
- Python 3 for the sample-size example
- Access to an analytics source
- Roughly estimated traffic volume
- A clear user problem
How does the sample-size example work?
The skill includes an illustrative calculation for a binary metric changing from 10% to 11%, with 50/50 allocation, two-sided alpha of 0.05 and power of 0.80. The script uses a large-sample approximation based on Cohen's proportion effect size and prints the per-variant sample. For these assumptions the expected output is 14745 observations per variant. Those baseline and variant rates are illustrative assumptions, not measured data, so replace them with your own numbers.
The example assumes independent units, one binary outcome, a fixed horizon and no multiplicity adjustment. It is not suitable for clustered or repeated observations, sequential decisions or continuous revenue metrics.
from math import asin, ceil, sqrt
from statistics import NormalDist
baseline, variant = 0.10, 0.11 # illustrative assumptions, not measured data
alpha, power = 0.05, 0.80
h = abs(2 * asin(sqrt(variant)) - 2 * asin(sqrt(baseline)))
z = NormalDist()
per_variant = ceil(2 * (z.inv_cdf(1 - alpha / 2) + z.inv_cdf(power)) ** 2 / h ** 2)
print(per_variant)What tracking checks does the skill require?
Before the execution readiness gate, the skill asks you to verify tracking concretely. This covers five checks, each of which should be recorded.
- Event firing: trigger each event the metrics depend on on staging or a debug page, and record the observed delay.
- Variant attribution: confirm the variant assignment ID is attached to every event, comparing a sample of 5+ events per variant in the raw event view.
- De-duplication: confirm that reloading a page does not double-count events, using a stable event or transaction ID.
- Sample randomization: check sample-ratio mismatch against the configured allocation with a pre-specified statistical test and adequate records.
- Guardrail pipeline: each guardrail metric has a working dashboard or alert before launch.
What rules does the skill enforce during and after the test?
During the test the agent monitors technical health and documents external factors. It must not stop early because results look good, change variants mid-test, add traffic sources or redefine success criteria. After the test, it should not generalize beyond the tested population, claim causality beyond the tested change, or override a guardrail failure. A guardrail failure means do not ship, even if the primary metric wins.
- Stopping early on good-looking results is ruled out.
- Variants stay fixed mid-test.
- Guardrail failures block shipping.
- Each test gets a written record: hypothesis, variants, metrics, sample size planned vs achieved, results, decision, learnings and follow-up ideas.
What are the safety notes?
The skill runs no scripts and uses no network access; it is a set of instructions. The sample-size example is code you can run yourself with Python 3. The skill also states that it cannot guarantee valid results by itself. It refuses to proceed when the baseline rate is unknown and cannot be estimated, traffic is insufficient to detect the MDE, the primary metric is undefined, several variables change without proper design, or the hypothesis cannot be stated clearly. In those cases it explains why and suggests next steps.
- Runs scripts: no
- Uses network: no
- Licence: MIT
- Source file: SKILL.md
What people use it for
- Planning a signup or checkout change for a website with a documented hypothesis and primary metric.
- Writing a reviewable test plan that analysts can check before launch.
- Checking that event tracking, variant attribution and de-duplication work before an app experiment starts.
- Deciding whether a low-traffic idea should be tested at all, or handled with usability research instead.
Frequently asked questions
How do I install the ab-test-setup agent skill?
Download the ZIP from the directory page and unzip it. Put the ab-test-setup folder in ~/.claude/skills/ for all projects or in a project's .claude/skills/ folder, or upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
Is the ab-test-setup skill free to use?
The skill is published under the MIT licence, as stated on the directory page. Check the licence file in the repository for the full terms.
What do I need to run the ab-test-setup skill?
The skill itself is instructions only. The sample-size example needs Python 3 and references the statsmodels library, and you need access to an analytics source and a rough traffic estimate.
Does the skill run code or access the internet?
According to the directory listing, it runs no scripts and uses no network access. The Python sample-size example is separate code you run yourself.
Can this skill guarantee my A/B test results are valid?
No. The skill states that its procedure cannot guarantee validity by itself. It provides a structured checklist and guidance, not a verified statistics calculator.
When should I not use this skill?
Use something else for low-traffic launches or qualitative research. The skill says a randomized comparison is not appropriate there and recommends usability research or descriptive measurement instead.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
The ab-test-setup skill suits product, growth and analytics teams who want a disciplined, reviewable checklist for A/B tests, with predeclared metrics, sample size and tracking checks. It is not a statistics tool and does not guarantee valid results, so teams needing computed analysis or low-traffic guidance should use a dedicated calculator or usability research instead.
Download the skill