Data Scientist interview question
How do you validate a machine learning model?
Validate a model with a split that mirrors future use, prevent entity and time leakage, choose metrics tied to the decision, compare a baseline, estimate uncertainty, inspect segments, and reserve a final untouched test set.
Direct answer
What a strong answer covers
How it works
Validation estimates how the model will perform on the data it will actually see, so the split must mirror deployment: by time when the model predicts the future, by entity when the same customer must not appear on both sides, and with preprocessing fitted inside each fold so nothing leaks. The metric follows the decision the model supports, not the default of the library, and it is reported with a confidence interval and against a simple baseline so a gain is known to be real. Performance is inspected by segment because an average hides groups where the model fails. A final test set is touched once.
Example
A churn model trained on data through June was validated on July through September with the scaler fitted only on the training window; AUC of 0.81 was compared to a baseline rule of days-since-last-login at 0.74, with a bootstrap interval of 0.79 to 0.83. By segment, precision on enterprise accounts was half that on self-serve, which changed how the sales team used the scores.
Common mistakes
- Using a random split on time-series data and reporting inflated results.
- Choosing accuracy on a 3 percent positive class and celebrating 97 percent.
- Iterating on the test set until the number looks good.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.