ab-testing

This skill serves as an expert guide for planning, designing, and executing A/B tests and broader growth experimentation programs. It helps users define test contexts, understand baseline performance, and identify constraints before starting. The skill outlines core principles for effective experimentation, emphasizing the importance of a clear hypothesis, testing single variables, maintaining statistical rigor, and measuring metrics directly tied to business value.

It offers a structured hypothesis framework with examples for crafting strong, testable predictions. Users can leverage this skill to determine appropriate sample sizes, utilizing quick reference tables and external calculators. It also guides the selection of primary, secondary, and guardrail metrics to ensure comprehensive evaluation of test outcomes.

The skill details various test types, including A/B, A/B/n, MVT, and Split URL tests, advising on the traffic needed for each. It helps users build a systematic experimentation practice, moving beyond simple comparisons to a data-driven approach for optimizing performance.