Product Manager interview question
How do you design a product A/B test?
Define the hypothesis, eligible population, randomization unit, primary metric, guardrails, minimum detectable effect, power, duration and stopping rule before launch; validate assignment and analyze uncertainty and segments carefully.
Direct answer
What a strong answer covers
How it works
The design is fixed before launch so results cannot be reinterpreted afterwards. It states the hypothesis as a predicted change in a specific metric for a specific population, the eligibility rules, the unit of randomisation such as user or account, the primary metric and the guardrails that must not worsen, the minimum effect worth detecting, and from that the sample size, duration in whole weeks to cover weekly cycles, and the stopping rule. Assignment is verified for balance and sample ratio, the analysis reports the effect with an interval, and segments are examined only as planned to avoid discovering a winning subgroup by chance.
Example
Hypothesis: showing delivery dates on the product page raises add-to-cart rate for logged-in users by at least 3 percent relative. Randomised by user, 50/50, primary metric add-to-cart, guardrails checkout completion and page load time, power analysis requiring 180,000 users per arm, two full weeks. A sample ratio check on day one caught a caching bug that showed the variant to returning users only, and the test was restarted.
Common mistakes
- Stopping the test the first day the result looks significant.
- Choosing the winning metric after seeing which one moved.
- Randomising by session so one user sees both variants and the comparison is contaminated.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.