Skip to content
Get packs

A/B Test Analysis

Analyze A/B test results for significance and get a ship, extend, or stop recommendation

Source Download skill
Repository
phuryn/pm-skills
Stars
27,616
Licence
MIT
Files
1 file

What it does

Once installed, the agent reviews A/B test results more rigorously: it checks whether the test was set up properly (sample size, duration, traffic split), calculates conversion rates, lift, p-values and confidence intervals, and checks guardrail metrics like revenue or page load time. It then produces a summary table and a recommendation such as ship, extend, stop, or investigate. If you provide raw data files, it is directed to write and run Python scripts for the calculations.

Great for

  • Product managers reviewing whether an experiment reached statistical significance
  • Analysts who want a consistent checklist for interpreting split test data
  • Teams deciding whether to roll out a variant to all users

Skip it if

  • Experiments with very small samples or no pre-defined hypothesis, where the statistics will be unreliable and results may mislead, so a statistician should review
  • Decisions with major financial or safety consequences that rely solely on an AI-generated recommendation without expert validation
  • Non-A/B designs such as multivariate or sequential tests, which this skill does not explicitly cover

Try asking

  • “Analyze these A/B test results for our checkout button change and tell me whether to ship it”
  • “Here is a CSV export of our onboarding experiment. Check if it reached significance and flag any guardrail issues”
  • “Did our pricing page test reach significance? Give me a markdown summary with a recommendation”
Step-by-step guide: how to use A/B Test AnalysisInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 1 original files, the MIT licence and an attribution note.

    ab-test-analysis.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip ab-test-analysis.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip ab-test-analysis.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Provide the hypothesis, primary metric, guardrail metrics, test duration and traffic split up front so the checks can be completed
  • Share raw data rather than only summary numbers so the agent can verify sample ratio mismatch and recalculate statistics
  • Treat the recommendation as a structured starting point and confirm the statistics with your analytics team for high-stakes decisions

What's inside

  • SKILL.md

Redistributed unmodified from phuryn/pm-skills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.