How do you manage feature pipelines?

Direct answer

What a strong answer covers

How it works

A feature pipeline is a data product: each feature has an owner, a schema, a documented definition, and point-in-time semantics that state exactly which events count as of a given timestamp. Transformations are versioned code, not notebook cells, and run through the same orchestrator as other data jobs with freshness and distribution checks on their sources. Offline materialisation for training and online materialisation for serving use one definition, backfills are idempotent so a rerun produces identical rows, and lineage connects each feature to the models that consume it so a change can be impact-assessed.

Example

Customer features are defined in a feature repository: each has an owner, an entity key, a timestamp column, and a SQL or Python transform; a nightly job materialises them to the offline store with the event timestamp, and an hourly job pushes the latest values to Redis. When a definition changed from 30-day to 28-day windows, lineage showed three models consumed it, each retrained on the new version before serving switched.

Common mistakes

• Computing features in a notebook and copying the code into production by hand.
• Backfilling with a job that appends and produces duplicate rows on rerun.
• Changing a feature definition without knowing which models depend on it.

A useful answer structure

Answer from your own work.

Do not memorise a script: use only examples you can explain honestly in follow-up questions.