ab-testing turns your agent into an experimentation partner. It starts from a hypothesis, picks primary, secondary and guardrail metrics, works out sample size and duration, and designs variants that change one thing at a time.
Once a test is running it helps read the result: significance, the peeking problem, whether to stop, and which version won. It also covers a growth experimentation program with ICE prioritization, an experiment playbook and a cadence.
What You Get
- A hypothesis written in the because / we believe / will cause / we’ll know when format
- Sample size and duration guidance with a quick reference table
- Primary, secondary and guardrail metric choices for the test
- A pre-launch checklist and rules for what to do during the run
- An analysis checklist and a documentation template for each experiment
Common Use Cases
- Deciding whether a pricing page change is worth a test and how long to run it
- Checking if a result is statistically valid before shipping the winner
- Building a prioritized experiment backlog with ICE scoring
- Reviewing a running test for peeking and early-stopping mistakes
What Makes It Unique
It treats judgement calls as first-class: is the result real, can we stop, is this worth testing at all. It also covers opt-in elements by measuring the exposed group, and extends from single tests to an experiment loop with a playbook, and it hands off tracking to analytics and page fixes to cro.
Under The Hood
The skill is prose only. Its SKILL.md and two reference files (a sample size guide and test templates) are loaded into context, plus an evals file the server leaves out. It runs no commands and calls no services; it may read a local product marketing context file.