Data Engineer interview question
How do you orchestrate data pipelines?
Model pipelines as idempotent tasks with explicit dependencies, inputs, outputs, retries, timeouts, backfills, concurrency limits, and ownership; persist run state and expose data-quality and freshness signals.
Direct answer
What a strong answer covers
How it works
An orchestrator such as Airflow or Dagster runs a graph of tasks with explicit dependencies on a schedule or on data arrival. Each task is idempotent so a retry or a backfill produces the same result, declares its inputs and outputs so lineage and freshness can be derived, and has a timeout and a retry policy suited to its failure modes. Concurrency limits protect shared systems, run state is persisted so a failure resumes rather than restarts, and each pipeline has an owner and alerting tied to the data it produces, not just to whether the task exited zero.
Example
The revenue pipeline is a DAG: extract billing, extract CRM, both feed a dbt run, which feeds quality tests, which gate the publication of marts. Each extract writes a dated partition and can rerun for any past day; a backfill of last month runs 30 dated instances with a concurrency cap of 4 so the source API is not overloaded. The publication task alerts finance only when tests pass and freshness is confirmed.
Common mistakes
- Writing tasks that append on each run so a retry doubles the data.
- Chaining tasks through implicit timing instead of declared dependencies.
- Alerting on task failure only and missing a task that succeeded with zero rows.
A useful answer structure
Answer from your own work.
Do not memorise a script: use only examples you can explain honestly in follow-up questions.
Open the desktop workspace and rehearse a clear answer in your own words.