Data Analyst interview question
What does a p-value tell you?
A p-value is the probability, assuming the null hypothesis and model assumptions are true, of observing a result at least as extreme as the data. It is not the probability that the hypothesis is true.
Direct answer
What a strong answer covers
How it works
A p-value quantifies how surprising the observed data would be if the null hypothesis were true and every modelling assumption held. A small value says the data are unusual under the null; it says nothing about how likely the alternative is, how large the effect is, or whether the result matters. The threshold of 0.05 is a convention, not a law, and running many tests inflates the chance of a small p-value by luck. A p-value is reported together with the effect size and a confidence interval, and interpreted against the sample size and the practical decision at stake.
Example
An A/B test reports p = 0.03 for a 0.2 percent lift in conversion; the result is statistically unlikely under no difference, but the confidence interval spans from 0.01 to 0.4 percent and the lift may be too small to justify the engineering cost. A dashboard that checked twenty metrics daily flagged one at p < 0.05 every week purely from multiple comparisons.
Common mistakes
- Reading p = 0.03 as a 97 percent chance that the change works.
- Peeking at results repeatedly and stopping when p dips below 0.05.
- Reporting significance without the effect size or interval.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.