How do you roll back an ML deployment?

Direct answer

What a strong answer covers

How it works

Rollback is designed before the first deployment. The release unit is the model, its feature contract version, its preprocessing, and its runtime configuration, versioned together so restoring the previous one restores a coherent system. Traffic is routed through a switch that can move percentages instantly, the last healthy release stays deployed or is restorable in minutes, and automated rollback fires on service guardrails such as error rate, latency, and prediction distribution shifts. Quality regressions that need delayed labels are handled by a separate, human-owned decision with pre-agreed thresholds, and the feature pipeline keeps producing the old contract until the new release is confirmed.

Example

During a rollout, the new model’s share of high-risk predictions tripled within an hour; the automated guardrail on prediction distribution routed traffic back to the previous version before a human looked. The postmortem found a feature whose scale had changed with the new contract. The rollback worked because the old feature version was still materialised, which the runbook had required.

Common mistakes

• Deploying a new feature contract and decommissioning the old one on the same day.
• Having only a manual rollback that needs a deploy pipeline to run.
• Setting no prediction-distribution guardrail so a wrong model looks healthy to the service monitors.

A useful answer structure

Answer from your own work.

Do not memorise a script: use only examples you can explain honestly in follow-up questions.