How do you monitor a deployed machine learning model?

Direct answer

What a strong answer covers

How it works

Monitoring a deployed model has four layers. The service layer is like any API: latency percentiles, error rate, saturation, and fallback activation. The data layer checks that inputs match the expected schema, that missing rates and distributions stay near the training baseline, and that upstream freshness holds. The prediction layer tracks the output distribution, confidence, and the share of predictions that trigger the action, so a model that suddenly flags everything is caught within minutes. The outcome layer joins delayed labels to compute accuracy, calibration, and business impact by model version and segment. Alerts route to the owning team with the version attached.

Example

A lead-scoring model’s dashboard shows p99 latency at 40 ms, a feature drift score per input, the daily share of leads scored above the threshold, and, with a 30-day lag, precision by version and by region. An alert fired when the share of high scores doubled in a day; the cause was a CRM change that filled a previously sparse field with a default, caught by the data layer before sales noticed.

Common mistakes

• Monitoring only uptime and treating a wrong model as a healthy one.
• Skipping outcome monitoring because labels arrive late.
• Aggregating across versions during a rollout so the new model’s problems are diluted.

A useful answer structure

Answer from your own work.

Do not memorise a script: use only examples you can explain honestly in follow-up questions.