How do you test an ML system?

Direct answer

What a strong answer covers

How it works

An ML system is tested at each layer where it can silently go wrong. Data tests check schemas, ranges, and freshness. Feature tests assert that transformations are deterministic and point-in-time correct on fixture data. Training tests run a tiny end-to-end fit and check that the pipeline produces the same metrics for the same inputs and seeds. Model tests assert invariants such as monotonic behaviour where the domain demands it, minimum performance on a golden set, and per-segment thresholds. Serving tests verify the request and response contract, latency under load, and fallback activation. Finally, a canary in production compares outcomes against the previous version on live traffic.

Example

The pricing model repo runs: a schema test on a fixture batch, a test that the same fixture through the feature code yields identical output across commits, a training smoke test on 1,000 rows with a fixed seed, an invariant test that a higher demand signal never lowers the recommended price, a golden-set threshold of 0.9 on a stored evaluation slice, and a contract test hitting the serving container. A canary at 5 percent must show no drop in conversion for 48 hours.

Common mistakes

• Testing only the code and never the data or the model behaviour.
• Comparing metrics against a moving baseline so regressions hide in noise.
• Skipping serving-contract tests and discovering a renamed field in production.

A useful answer structure

Answer from your own work.

Do not memorise a script: use only examples you can explain honestly in follow-up questions.