Software Engineer interview question
How do you debug a production latency regression?
Confirm the regression with percentile metrics, correlate it with deploys and traffic, use traces to isolate the slow dependency or code path, inspect saturation and query plans, mitigate safely, then validate against the same baseline.
Direct answer
What a strong answer covers
How it works
First confirm the regression is real with percentile metrics over a stable window and identify when it started and which endpoints, regions, or customers it affects. Correlate that moment with deploys, config changes, traffic shifts, and dependency incidents. Distributed traces from slow requests show which span grew; saturation metrics show whether a pool, a CPU, a disk, or a lock is the constraint; slow-query logs and plans show whether the database changed its mind. Mitigate with the least risky action, often a rollback or a flag, then fix properly and verify against the same baseline.
Example
p95 doubled at 14:10; the deploy log showed a release at 14:05 that added a feature flag check calling a config service on every request. Traces showed a new 80 ms span per call, and the config service’s CPU was pegged. Rolling back restored p95 in three minutes; the fix cached the flag in process with a 30 s refresh and shipped the next morning with a load test proving it.
Common mistakes
- Restarting services as a first move and destroying the evidence.
- Optimising code without a trace showing that code is the slow part.
- Declaring victory from the average when the p99 is still broken.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.