Skip to content
SkillModel-led2,830 tokens in use

ab-testing

Plans, runs and reads A/B tests: hypotheses, sample size, guardrail metrics, and calls on whether a result is real or a test can stop.

61,709 stars on the repository MIT Licensecoreyhaines31/marketingskills · skills/ab-testing
npx skills add coreyhaines31/marketingskills --skill ab-testing
View source

Install command as written in the repository's README

Skill label

measured from its files

What installing ab-testing costs

76tokens idle
2,830tokens in use
7,019tokens, up to

Idle is what sits in context while the skill is installed. In use is its instructions, loaded when it runs. Up to adds the files its instructions link. Counts are approximate, made offline with cl100k_base.

Model dependence Model-led

Script-ledMixedModel-led

Mostly instructions, so how well it works depends on the model that follows them.

This describes the skill's structure, how much of it is scripts. It does not say how well it works.

Touches

Nothing detected: no shell, network, file writes, git or credentials in its files.

Declared tags come from the skill's frontmatter; detected ones from its scripts, its code blocks and the commands it names in inline code. This is detected, not audited.

How it scores

Quality

How well the skill is written, counted from its own files: its frontmatter, its steps, examples, stated output and constraints, its size and whether every file it ships is one it names.How we score it →
Excellent
  • Frontmatter12/15
  • Structure19/25
  • Examples15/15
  • Output10/15
  • Size15/15
  • Resources15/15

Security

Starts at 100 and loses points for each thing in the skill's files that a reader should look at first.How we score it →
Excellent

No findings.

Repository health

Pre-computed score based on development activity, maintenance, community, maturity, and trend momentum.How we score it →
Excellent
  • Development activity100
  • Maintenance100
  • Community76
  • Maturity32
  • Momentum40

ab-testing turns your agent into an experimentation partner. It starts from a hypothesis, picks primary, secondary and guardrail metrics, works out sample size and duration, and designs variants that change one thing at a time.

Once a test is running it helps read the result: significance, the peeking problem, whether to stop, and which version won. It also covers a growth experimentation program with ICE prioritization, an experiment playbook and a cadence.

What You Get

  • A hypothesis written in the because / we believe / will cause / we’ll know when format
  • Sample size and duration guidance with a quick reference table
  • Primary, secondary and guardrail metric choices for the test
  • A pre-launch checklist and rules for what to do during the run
  • An analysis checklist and a documentation template for each experiment

Common Use Cases

  • Deciding whether a pricing page change is worth a test and how long to run it
  • Checking if a result is statistically valid before shipping the winner
  • Building a prioritized experiment backlog with ICE scoring
  • Reviewing a running test for peeking and early-stopping mistakes

What Makes It Unique

It treats judgement calls as first-class: is the result real, can we stop, is this worth testing at all. It also covers opt-in elements by measuring the exposed group, and extends from single tests to an experiment loop with a playbook, and it hands off tracking to analytics and page fixes to cro.

Under The Hood

The skill is prose only. Its SKILL.md and two reference files (a sample size guide and test templates) are loaded into context, plus an evals file the server leaves out. It runs no commands and calls no services; it may read a local product marketing context file.

Join founders building with open source

Opinionated takes, migration guides, cost-saving tips, and insights from the open source ecosystem.

Subscribe on SubstackJoin 750+ subscribers