How to use the A/B Test Setup agent skill (ab-testing)

What is an agent skill and what does this one do?

An agent skill is a folder of instructions and reference files that an AI agent can load when a task matches it. The ab-testing skill turns the agent into an experimentation helper when you want to compare two approaches and measure which performs better.

Once installed, the agent can help you write a clear hypothesis, pick one primary metric plus guardrail metrics, estimate how many visitors you need and how long to run the test, and avoid common mistakes such as stopping early. It can also explain statistical results, including p-values and confidence intervals, in plain terms.

The files are instructional guidance and reference tables. They do not set up any testing tool for you.

Install the ab-testing skill

Download the ZIP from the skill's page on this site and unzip it. You have two places to put the folder. For personal use across projects, place it in ~/.claude/skills/. For one project only, place it in that project's .claude/skills/ folder.

You can also upload the skill in the Claude apps. Go to Settings, then Capabilities, then Skills, and upload the ZIP there.

• Personal: ~/.claude/skills/ab-testing
• Single project: .claude/skills/ab-testing inside your project folder
• Claude apps: Settings → Capabilities → Skills → upload the ZIP

Ask for it with natural requests

You do not need a special command. The skill is triggered by what you ask. Describe the comparison you want to measure, and the agent should pick it up. Phrases the skill is written to respond to include "A/B test," "split test," "test this change," "which version is better," "test two versions," "statistical significance," "how long should I run this test" and "hypothesis."

For program-level work, phrases such as "experiment backlog," "ICE score" and "experiment playbook" are also listed. If your request is about tracking setup, the skill says to use an analytics skill instead. For page-level conversion work, it points to a cro skill.

• "Should I test a shorter headline against my current one on the pricing page?"
• "How long should I run this test if my signup page gets 2,000 visitors a week?"
• "Help me write a hypothesis for a larger button on the checkout page."
• "Set up an experiment backlog and rank ideas with an ICE score."

Build a sound test with the skill

The skill asks for three things before designing a test: what you are trying to improve and the change you are considering, your current baseline conversion rate and traffic volume, and any technical or timeline constraints. If a file named .agents/product-marketing.md or .claude/product-marketing.md exists, the agent reads it first.

The skill's hypothesis structure is: because of an observation or data, we believe a change will cause an expected outcome for an audience, and we will know this is true when specific metrics move. It also insists on testing one thing per test and on defining a primary metric, secondary metrics and guardrail metrics.

• Hypothesis: Because [observation/data], we believe [change] will cause [expected outcome] for [audience].
• Primary metric: the single metric that will decide the test.
• Guardrail metrics: things that must not get significantly worse, such as refund rate or support tickets in the pricing example.
• Test types covered: A/B, A/B/n, multivariate (MVT) and split URL.

Estimate sample size and duration

The reference file references/sample-size-guide.md holds the sample size tables and duration rules. The main quick-reference table is given per variant, at 95% confidence (two-sided) and 80% power, with relative lifts of 10%, 20% and 50% across baselines of 1%, 3%, 5% and 10%. Lift is relative: going from 3% to 3.6% is a 20% lift.

The skill links two online calculators, Evan Miller's and Optimizely's. Use them to check numbers before you commit to a run length. The skill notes that the agent may still make calculation mistakes.

• Required inputs: baseline conversion rate, minimum detectable effect, significance level (usually 0.05) and power (usually 80%).
• Example from the table: a 5% baseline with a 20% lift needs about 8.2k visitors per variant.

Read results without overclaiming

The skill explains that a p-value is the probability of a result at least as extreme as the one observed, assuming the null model and its assumptions. It is not the probability that the result is random or that a variant will win again. A 95% confidence interval should be reported alongside the effect size, not described as a 95% chance the true effect is inside it.

The skill's analysis checklist asks whether you reached the planned sample size, whether the result is significant with its interval, and whether the effect is meaningful compared with the minimum detectable effect. It also states that statistical significance alone does not justify shipping a variant.

• Stopping early (peeking) can produce false positives. Commit to the sample size first.
• Do not change variants or add traffic sources during the test.
• Sequential and Bayesian tools have their own interpretation; do not treat every platform's confidence score as a standard p-value.

Safety notes: scripts and network

The skill requires nothing to be installed beforehand. It does not run scripts and does not use the network, according to its listing. Its files are text guidance and tables. Because it does not connect to your analytics or testing tools, you must still set up tracking and variants yourself, or use a separate analytics or testing tool.