Data Scientist interview question
How do you choose between candidate models?
Choose candidate models by out-of-sample performance on the business-relevant metric, uncertainty, calibration, robustness by segment, latency, explainability, maintenance, and cost—not by the best single cross-validation score.
Direct answer
What a strong answer covers
How it works
The best cross-validation score is a starting point, not a decision. Candidates are compared on the metric tied to the business decision, with confidence intervals so a 0.3 percent difference is recognised as noise. Beyond that, the choice weighs calibration when probabilities drive actions, robustness across segments and over time, latency and cost at serving volume, explainability requirements from regulators or users, and how much maintenance the feature pipeline demands. A simpler model that is within the interval of the best one often wins because it is cheaper to run, explain, and monitor.
Example
Three candidates for a credit-limit model had AUCs of 0.842, 0.845, and 0.847 with overlapping intervals; the gradient-boosted one was best but required 40 features from a real-time source, while the logistic model used 12 batch features and produced reason codes the compliance team needed. The logistic model shipped, with a plan to revisit if the segment analysis showed decay.
Common mistakes
- Picking the leaderboard winner by a margin smaller than the noise.
- Ignoring serving cost until the model cannot meet the latency budget.
- Selecting a black-box model for a decision that legally requires an explanation.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.