Instructions and covered skills
Read each scenario carefully before selecting an answer. Focus on the stated metric, population, time frame, and decision context. Do not infer details that are not provided in the question. Keep notifications off and avoid switching between tasks while completing the test. Use consistent statistical reasoning rather than relying on a single percentage change. Review each selected response for whether it addresses the experimental question directly.
Key Areas
This assessment covers the statistical judgment required to run and interpret controlled A/B tests. Candidates work with null and alternative hypotheses, p-values, confidence intervals, statistical power, minimum detectable effects, and error rates. They should understand that random assignment makes treatment groups comparable on average and that departures from expected traffic allocation can signal implementation or measurement problems.
The assessment also addresses experiment design choices. This includes selecting one primary outcome, defining the unit of randomization, establishing eligibility rules, estimating sample requirements, and setting a decision rule before reviewing results. Strong performance requires recognizing threats such as repeated significance checks, metric definition changes, contaminated treatment exposure, missing conversion events, and novelty effects.
Interpretation is central. Candidates must distinguish relative uplift from absolute impact, statistical evidence from business value, and aggregate results from subgroup findings. They should know when a confidence interval suggests meaningful uncertainty, why multiple comparisons can increase false positive findings, and how guardrail metrics can prevent a local improvement from causing broader harm.
Recommended Preparation
Prepare by reviewing the lifecycle of a controlled experiment: state a decision, define a primary metric, write hypotheses, select an assignment method, calculate a sample target, run a quality check, analyze the planned comparison, and document the conclusion. Practice converting between conversion counts, conversion rates, absolute percentage-point changes, and relative changes. Review the interpretation of p-values and confidence intervals without treating either as a guarantee of replication or business success.
It is also useful to examine realistic experiment readouts. Identify the target population, exposure period, denominator, event definitions, treatment allocation, and guardrail outcomes before drawing conclusions. When reviewing a result, ask whether the analysis follows the original plan, whether the data collection process appears valid, and whether the observed effect is large enough to justify the operational cost or risk of rollout.