Machine Learning Engineer interview question
How do you scale model inference?
Scale inference by profiling preprocessing and model compute, batching compatible requests, selecting efficient runtimes and hardware, autoscaling on queue and latency, bounding concurrency, caching safe results, and applying load shedding.
Direct answer
What a strong answer covers
How it works
Scaling begins with a profile of one request: preprocessing time, feature fetch time, and model compute are often in surprising proportions, and the biggest one is the target. Compute scales through dynamic batching, quantised or distilled models, efficient runtimes such as ONNX Runtime or TensorRT, and accelerators matched to the model size. Capacity scales through autoscaling on queue depth and latency rather than CPU, with bounded concurrency per replica so tail latency stays flat. Caching helps when identical inputs repeat, load shedding protects the tail when demand exceeds capacity, and every change is measured against the same p99 under a replayed traffic pattern.
Example
A text classification endpoint spent 70 percent of its latency in tokenisation on the Python side; moving to a compiled tokenizer and batching 32 requests per GPU call tripled throughput before touching the model. Autoscaling on queue depth with a maximum of 8 in-flight requests per replica kept p99 under 80 ms during a 10x traffic spike, while a 5 percent cache hit rate on repeated inputs was found not worth the complexity.
Common mistakes
- Buying larger GPUs before profiling where the time actually goes.
- Autoscaling on CPU utilisation for a GPU-bound service.
- Letting unbounded concurrency per replica turn a spike into a latency collapse.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.