Data Scientist interview question
How do you design a data-science experiment?
Define the causal question, population, treatment, control, primary metric, guardrails, effect size, power, randomization unit, duration, and analysis plan before looking at outcomes; track assignment and interference risks.
Direct answer
What a strong answer covers
How it works
An experiment is designed before any data is seen. It states the causal question and the population, what the treatment and control experience, the unit of randomisation, which must match the unit of analysis to avoid interference, the primary metric and the guardrails, the minimum effect worth detecting, the sample size and duration that give adequate power, and the analysis plan including how segments will be examined. Assignment is logged and checked for balance, and risks such as users in both arms, network effects, or novelty are named with a plan for each.
Example
A test of a new recommendation algorithm randomised by user, not by session, because a user seeing both versions would contaminate the comparison; the primary metric was 14-day purchase rate, guardrails were return rate and page latency, and power analysis at a 2 percent relative lift required 400,000 users per arm over three weeks. A sample ratio mismatch check on day two caught a bug assigning 52 percent to control.
Common mistakes
- Choosing the primary metric after seeing which one moved.
- Randomising by session when the same user can see both treatments.
- Running until significance appears instead of for the planned duration.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.