A/B Testing (Experimentation)

Valid A/B testing requires: a clearly defined hypothesis ("Changing button text from 'Start Free Trial' to 'Try Free for 14 Days' will increase click-through rate by 10%"); a single measured metric (the "primary metric," agreed before the test starts); a pre-calculated sample size (use a power analysis calculator with desired statistical power of 80% and significance level of 0.05); a defined runtime (never stop an A/B test early because it looks like it is working — this dramatically inflates false positive rates); and random assignment of users to variants (not by day of week or any correlated variable). Product Ops builds the experimentation framework — the tooling, documenting the testing SOP, conducting power analyses, and running the post-experiment statistical analysis.

The most dangerous A/B testing mistake is "peeking" — checking results before the predetermined sample size is reached and stopping early when the result looks positive. This exploits natural variance and produces false positives at rates far exceeding the stated significance level. Other common pitfalls: testing too many variants simultaneously (dilutes sample per variant, extends runtime, and complicates interpretation); measuring secondary metrics instead of the primary metric defined upfront; failing to account for novelty effects (users engage more with anything new for the first week before returning to baseline); and not segmenting results (the winning variant overall may be losing for your most valuable customer segment). Product Ops maintains an A/B testing log documenting every experiment, its hypothesis, results, and decision — creating an institutional memory of what the team has learned.

A mature experimentation program requires infrastructure, culture, and governance. Infrastructure: experimentation platform (Statsig, Optimizely, or feature flag tool with built-in experimentation like LaunchDarkly), reliable analytics event tracking to measure primary metrics, and a statistical analysis template. Culture: a team norm that decisions may be challenged with "what experiment could we run to validate this?" and leaders who follow experiment results even when they contradict intuition. Governance: an experiment backlog prioritized by expected impact and feasibility, a review process that validates statistical validity before reading results, and a results database making findings accessible to all PMs. Product Ops owns all three dimensions and reports monthly on the number of experiments run, the percentage with statistically significant results, and the cumulative impact (metric improvements from winning experiments).