Pipelines & scenarios · Core Pipeline Concepts

What is a data pipeline?

Easypipelinearchitecturemental modelmedallionbasics

Interview question

What is a data pipeline?

How to answer

The short answer

A is the end-to-end path that moves data from sources to consumers, with transforms, quality checks, schedules, and ownership along the way.

It is not just "a Spark job." It is the whole flow: extract, land, clean, model, serve, monitor.

Big mental model (keep this diagram in your head)

Pipeline vs DAG

  • Pipeline = business data flow (what data moves and why).
  • = orchestration graph (how tasks are scheduled and depended).

Fresher example

Nightly ecommerce pipeline: pull orders from Postgres → land Parquet in S3 → Spark cleans → dbt builds fct_orders → Looker dashboard refreshes.

Interview tip: Start with sources → land → transform → serve, then name orchestration, quality, and monitoring. That diagram alone shows you think in systems, not scripts.