Data Engineer interview question
How do you handle schema evolution?
Evolve schemas compatibly: add optional fields first, version contracts, let old and new producers coexist, validate consumers, backfill when required, and remove fields only after usage and retention windows expire.
Direct answer
What a strong answer covers
How it works
Schemas change constantly, and the goal is that producers and consumers can upgrade at different times. Additive changes, new optional fields with defaults, are safe; renames, type changes, and removals are breaking and are done as add-new, migrate, remove-old across separate releases. A schema registry with compatibility rules rejects a breaking change before it ships, consumers are written to ignore unknown fields, and the raw layer keeps the original bytes so a mistaken change can be reprocessed. Fields are removed only after monitoring shows no consumer reads them and the retention window for old data has passed.
Example
An event needed amount to move from a float to a decimal string. The producer added amount_decimal alongside amount for two months, consumers migrated one by one with the registry enforcing backward compatibility, a backfill populated amount_decimal for historical partitions, and amount was removed after usage metrics showed zero readers for 30 days.
Common mistakes
- Renaming a field in place and breaking every consumer at once.
- Changing a type and letting old data become unreadable.
- Removing a field the moment the new one exists, before consumers have migrated.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.