How do you monitor model drift?

Direct answer

What a strong answer covers

How it works

Drift monitoring compares what the model sees and produces in production against a versioned baseline captured at training time. Input monitoring checks schema, missing rates, and per-feature distributions with a distance measure and an alert threshold per feature. Output monitoring tracks the prediction distribution and confidence. When labels arrive later, performance and calibration are computed on that delayed cohort. Alerts are segmented so a shift in one region is visible, and every alert is first checked against the data pipeline, because a broken upstream join looks exactly like drift.

Example

A demand forecast model’s input monitor flagged that the promotion flag went from 8 percent to 0 percent of rows; the cause was a renamed column in the source table, not a change in the world, and the fix was in the pipeline. Two months later, prediction error on the delayed actuals rose steadily in one country after a competitor launch, and that triggered a retrain with recent data rather than a pipeline fix.

Common mistakes

• Alerting on every small distribution change and training the team to ignore alerts.
• Retraining automatically on drift without checking whether the data pipeline broke.
• Monitoring inputs only and never computing performance on delayed labels.

A useful answer structure

Answer from your own work.

Do not memorise a script: use only examples you can explain honestly in follow-up questions.