QA Automation Engineer interview question
How do you diagnose a flaky test?
Reproduce with repetition and captured traces, classify timing, shared state, environment, network or product race causes, remove sleeps, wait for observable conditions, isolate data, and fix ownership rather than blindly retrying.
Direct answer
What a strong answer covers
How it works
A flaky test is a test with a bug in it or a bug under it. Diagnosis starts by making it fail on demand: run it fifty times, in parallel with its neighbours, and with traces, screenshots, and logs captured on failure. The evidence sorts the cause into a class: timing, where the test asserts before the application has settled; shared state, where another test or a leftover record changes the outcome; environment, such as a slow CI runner or a network dependency; or a genuine product race that only shows under load. The fix matches the class, waiting on observable conditions rather than sleeping, isolating data, or opening a product bug, and the test is quarantined with an owner until it is fixed, never simply retried.
Example
A checkout test failed 3 percent of the time; fifty runs with tracing showed the assertion fired before a price recalculation request completed, and the visible total was momentarily stale. The fix replaced a 500 ms sleep with a wait for the network request to finish and the total to match. A different test that failed only after 14:00 UTC turned out to depend on a fixture with a hard-coded date, a shared-state class.
Common mistakes
- Adding retries so the flake is hidden instead of fixed.
- Fixing timing with longer sleeps that make the suite slower and still flaky.
- Assuming the test is at fault and missing a real race condition in the product.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.