Data Engineer interview question
How do you approach exactly-once data pipelines?
Exactly-once business effects require replayable input, deterministic processing, idempotent or transactional writes, stable event identities, and checkpointing. A broker's delivery guarantee alone does not make the whole pipeline exactly once.
Direct answer
What a strong answer covers
How it works
A message broker can promise at-least-once delivery, and some offer transactional exactly-once within their own boundary, but the business effect at the end of the pipeline depends on what the consumer does. The pipeline becomes effectively exactly-once when input can be replayed from a stable offset, processing is deterministic, every event carries a stable identity, and the sink write is idempotent or transactional with the checkpoint, so a crash between processing and committing the offset leads to a reprocess that changes nothing. Duplicates are treated as normal, not exceptional, and monitored.
Example
A payments stream is consumed from Kafka; each event has a payment_id, the sink is a table with payment_id as primary key and an upsert, and the consumer commits offsets only after the upsert succeeds. When a consumer restarted mid-batch, it reprocessed 200 events and the table did not change. A separate metric counts duplicate upserts so a broken producer that sends the same event thousands of times is noticed.
Common mistakes
- Trusting the broker’s exactly-once flag and writing with plain inserts.
- Committing the offset before the write and losing events on crash.
- Deriving the event identity from processing time so replays create new records.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.