ab-test-analysis walks an agent through evaluating an experiment: the hypothesis, the variant, the primary and guardrail metrics, and the setup checks. It calculates conversion rates, relative lift, a p-value and a 95% confidence interval, and maps the result to a clear recommendation. If you provide raw data, it generates and runs a Python script for the calculations.
What You Get
- A validation of sample size, test duration, randomization and novelty effects
- Conversion rates, relative lift, p-value and a 95% confidence interval for each metric
- A guardrail check that flags wins that hurt revenue, engagement or load time
- A recommendation of ship, extend, stop, investigate or do not ship
- A markdown summary with hypothesis, results table, reasoning and next steps
- A Python script for the statistics when you share raw data
Common Use Cases
- Checking whether a checkout experiment reached significance
- Deciding whether to extend an underpowered test
- Reviewing a split test where the primary metric won but a guardrail dropped
What Makes It Unique
It ties the statistics to a fixed decision table, so every result ends in one of five named recommendations. It also checks the setup before the numbers, including power, test duration and sample ratio mismatch, so a result from a flawed test is flagged early.
Under The Hood
The skill is one SKILL.md of instructions with no bundled scripts. It reads any CSV, Excel or analytics export you provide. When raw data is available, the agent writes and runs its own Python script to compute the z-test or chi-squared test and the confidence interval. The summary is saved as a markdown file.