Skip to content
SkillModel-led806 tokens in use

ab-test-analysis

Analyze A/B test results for significance, sample size and guardrail metrics, and get a ship, extend, stop or investigate recommendation.

28,201 stars on the repository MIT Licensephuryn/pm-skills · pm-data-analytics/skills/ab-test-analysis
claude plugin install pm-data-analytics@pm-skills
View source

Install command as written in the repository's README

Skill label

measured from its files

What installing ab-test-analysis costs

55tokens idle
806tokens in use

Idle is what sits in context while the skill is installed. In use is its instructions, loaded when it runs. Counts are approximate, made offline with cl100k_base.

Model dependence Model-led

Script-ledMixedModel-led

Mostly instructions, so how well it works depends on the model that follows them.

This describes the skill's structure, how much of it is scripts. It does not say how well it works.

Touches

Nothing detected: no shell, network, file writes, git or credentials in its files.

Declared tags come from the skill's frontmatter; detected ones from its scripts, its code blocks and the commands it names in inline code. This is detected, not audited.

How it scores

Quality

How well the skill is written, counted from its own files: its frontmatter, its steps, examples, stated output and constraints, its size and whether every file it ships is one it names.How we score it →
Good
  • Frontmatter12/15
  • Structure23/25
  • Examples9/15
  • Output7/15
  • Size15/15
  • Resources10/15

Security

Starts at 100 and loses points for each thing in the skill's files that a reader should look at first.How we score it →
Excellent

No findings.

Repository health

Pre-computed score based on development activity, maintenance, community, maturity, and trend momentum.How we score it →
Good
  • Development activity80
  • Maintenance76
  • Community60
  • Maturity32
  • Momentum40

ab-test-analysis walks an agent through evaluating an experiment: the hypothesis, the variant, the primary and guardrail metrics, and the setup checks. It calculates conversion rates, relative lift, a p-value and a 95% confidence interval, and maps the result to a clear recommendation. If you provide raw data, it generates and runs a Python script for the calculations.

What You Get

  • A validation of sample size, test duration, randomization and novelty effects
  • Conversion rates, relative lift, p-value and a 95% confidence interval for each metric
  • A guardrail check that flags wins that hurt revenue, engagement or load time
  • A recommendation of ship, extend, stop, investigate or do not ship
  • A markdown summary with hypothesis, results table, reasoning and next steps
  • A Python script for the statistics when you share raw data

Common Use Cases

  • Checking whether a checkout experiment reached significance
  • Deciding whether to extend an underpowered test
  • Reviewing a split test where the primary metric won but a guardrail dropped

What Makes It Unique

It ties the statistics to a fixed decision table, so every result ends in one of five named recommendations. It also checks the setup before the numbers, including power, test duration and sample ratio mismatch, so a result from a flawed test is flagged early.

Under The Hood

The skill is one SKILL.md of instructions with no bundled scripts. It reads any CSV, Excel or analytics export you provide. When raw data is available, the agent writes and runs its own Python script to compute the z-test or chi-squared test and the confidence interval. The summary is saved as a markdown file.

Join founders building with open source

Opinionated takes, migration guides, cost-saving tips, and insights from the open source ecosystem.

Subscribe on SubstackJoin 750+ subscribers