How to use the A/B Test Planner agent skill (ab-testing)

What the ab-testing skill makes the agent do

Once installed, the agent helps you plan controlled experiments. It asks what you want to improve and what change you are considering, then helps you write a testable hypothesis in a fixed structure: an observation, the change, the expected outcome, the audience and the metrics that will show whether it worked.

It also helps you choose a primary metric, secondary metrics and guardrail metrics, and recommends sample sizes and test duration using a named calculation method. Before launch it works through a pre-launch checklist and warns about common mistakes such as stopping a test early or misreading p-values.

• Turns a vague idea into a specific hypothesis
• Suggests primary, secondary and guardrail metrics
• Gives a sample size and duration approach with recorded inputs
• Provides a pre-launch checklist and analysis checklist
• Includes test plan and results templates for documentation

Install the ab-testing skill

Download the ZIP from the skill's page in our directory and unzip it. You can place the skill folder in one of two locations. For a personal skill available across your projects, use ~/.claude/skills/. For a skill limited to one project, put it in that project's .claude/skills/ folder.

If you use the Claude apps instead of a local setup, you can upload the skill under Settings → Capabilities → Skills.

• Personal: ~/.claude/skills/ab-testing/
• Project only: .claude/skills/ab-testing/ inside your project
• Claude apps: Settings → Capabilities → Skills, then upload

Trigger the skill with natural requests

You do not need a special command. The skill loads when your request is about planning, designing or implementing an A/B test or experiment, or building a growth experimentation program. Describe your situation in plain language and say what you want to achieve.

If you keep a product marketing context file at .agents/product-marketing.md or .claude/product-marketing.md, the skill reads it first so it does not ask for information you have already given.

• "Help me plan an A/B test for our pricing page."
• "What sample size do I need to test a new signup button?"
• "Give me a pre-launch checklist for this landing page experiment."
• "Set up an experimentation program for our growth team."

Examples of what you get back

For a pricing page test, the skill would suggest plan selection rate as the primary metric, time on page and plan distribution as secondary metrics, and support tickets and refund rate as guardrails. For a headline test, it would ask for the baseline conversion rate and the smallest lift worth detecting before recommending a sample size.

The skill's hypothesis example contrasts a weak idea (changing a button colour might increase clicks) with a strong one that names the evidence, the audience and the metric.

What you need and what the skill does not do

The skill lists no required software or packages. It runs no scripts and makes no network calls. It is mostly written guidance and reference tables rather than code, so the calculations it suggests are for you to confirm in a calculator or your testing tool.

It is not a formal statistical analysis pipeline. Use it for planning and checklists, not for analysing existing data rigorously.

• Requires: nothing listed
• Runs scripts: no
• Uses network: no
• Licence: MIT

Safety notes and tips

Because the skill runs no scripts and uses no network, the main risks are in the advice itself. Check any sample size against the calculators the skill links to, and treat its recommendations as a starting point for your own team's review. The skill also states that testing does not authorise unrequested production rollout or data sharing.

Tips: write the hypothesis before building anything, commit to a sample size in advance, and do not peek at results and stop early. Keep a test plan for every experiment so results can be compared later.

• Pre-commit the sample size and do not stop early
• Check for sample-ratio mismatch and missing events
• Keep variants accessible, honest and privacy-compliant
• Document every test, including inconclusive ones