Machine Learning Engineer interview question
How do you design a model serving system?
Design serving from latency, throughput, availability and freshness targets: version model and features together, validate inputs, batch where suitable, autoscale bounded workers, cache safely, expose fallback, and monitor predictions.
Direct answer
What a strong answer covers
How it works
A serving system starts from numbers: the latency budget per prediction, the requests per second at peak, the availability target, and how fresh features must be. The model and the preprocessing code are packaged and versioned together so the transformation at inference matches training exactly. Inputs are validated against a schema, requests are batched when the model benefits and the latency budget allows, workers autoscale within bounds, results that are safe to reuse are cached, and a fallback such as a previous model or a heuristic answers when the model is unavailable. Every prediction is logged with its version and features for monitoring.
Example
A ranking model serves 3,000 requests per second with a 50 ms p99 budget: features are fetched from an online store, a Triton server batches requests in 5 ms windows, the model container declares its feature schema and rejects mismatches, and if the store times out the service returns popularity-based ranking with a flag. Model version 14 handled 10 percent of traffic for a day before taking all of it.
Common mistakes
- Re-implementing preprocessing in the serving code and introducing training-serving skew.
- Having no fallback so a model outage becomes a product outage.
- Logging predictions without the model version, making incidents impossible to attribute.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.