A practical plan for defining, running, and evaluating a controlled experiment. Use before launching an experiment to align its hypothesis, audience, measures, safeguards, and decision rules.
OverviewIdentify the experiment, accountable owner, planned dates, and current status.
HypothesisState the change, the people it affects, the expected result, and the main measure.
BackgroundExplain the evidence or business context behind the experiment and why the proposed change is worth evaluating.
VariantsDescribe the current experience and the proposed version in enough detail to reproduce each one.
MetricsDefine the primary outcome, supporting measures, and guardrails before launch.
Sample size and durationRecord the assumptions and calculations used to set the required sample and run time.
Audience targetingSpecify who can enter, who must be excluded, and how eligible traffic is divided among variants.
Success criteriaSet decision rules before the experiment begins.
Risks and mitigationsList plausible ways the experiment could produce harm or misleading results, then assign an action to reduce each risk.
Monitoring planDescribe what the team will watch during the run, which thresholds prompt review, and when to stop or roll back.
Implementation notesCapture the technical details needed to launch and measure the experiment reliably.
ReferencesLink materials that help reviewers understand the rationale, design, or prior evidence.