Machine Learning Engineer interview question
When do you choose batch versus online inference?
Choose batch inference for predictable high-volume scoring when delayed results are acceptable; choose online inference for per-request freshness and low latency, accepting stricter availability, feature parity, and cost requirements.
Direct answer
What a strong answer covers
How it works
Batch inference scores a known population on a schedule and writes results to a table: it is cheap, easy to retry, and uses features that are already computed, and it is right when decisions are made hourly or daily, such as churn lists or nightly recommendations. Online inference answers a request in milliseconds with the freshest inputs, which is required when the input only exists at request time or when the decision changes what the user sees next, and it brings availability targets, online feature access, and always-on cost. Many systems precompute in batch and serve from a lookup, keeping online models for the few cases that need them.
Example
Product recommendations for the home page are computed nightly for every active user and stored in a key-value store, because the catalogue and user history change slowly. Search ranking runs online because the query is the main input. Fraud scoring at checkout is online with a 30 ms budget, and its batch counterpart runs nightly to catch patterns across many transactions the online model cannot see.
Common mistakes
- Building an online endpoint for a nightly report and paying for 24/7 availability.
- Serving stale batch scores where a user’s action a minute ago should change the answer.
- Ignoring that online serving needs features available at request time.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.