Machine Learning Engineer interview question
How do you control ML infrastructure costs?
Attribute cost by training run and endpoint, right-size accelerators, use spot capacity for restartable work, schedule and stop idle notebooks, batch inference, cache features, set quotas, and compare cost per useful prediction.
Direct answer
What a strong answer covers
How it works
Cost control starts with attribution: every training job and endpoint is tagged so spend is visible per team, project, and model, and the metric that matters is cost per useful prediction or per experiment, not total spend. Training uses spot or preemptible capacity with checkpointing, accelerators sized to the model rather than the largest available, and schedules that avoid idle time. Notebooks and dev endpoints shut down when idle. Inference batches where latency allows, caches repeated features, and right-sizes replicas from real traffic. Quotas per team stop a runaway grid search from consuming a month’s budget in a night.
Example
Cost attribution revealed that 40 percent of GPU spend went to notebooks left running over weekends and a hyperparameter sweep that re-ran nightly with unchanged results. Idle shutdown after 30 minutes, moving the sweep to spot instances with checkpoints, and replacing an online endpoint used once a day with a batch job cut monthly spend by 55 percent with no change in model quality.
Common mistakes
- Running training on on-demand instances when the job can checkpoint and restart.
- Leaving inference replicas sized for a peak that happens one hour a week.
- Having no per-team quota so one experiment can exhaust the shared budget.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.