Pipelines & scenarios · Core Pipeline Concepts

What is orchestration?

EasyorchestrationAirflowDAGschedulingdependencies

Interview question

What is pipeline orchestration, and what tools do data engineers use?

How to answer

The short answer

Orchestration is scheduling and coordinating pipeline tasks: order, dependencies, retries, concurrency, and alerts. The orchestrator usually does not do heavy compute; it starts and watches work elsewhere.

What orchestration gives you

  • Run B only after A succeeds
  • Retry A 3 times on failure
  • last 30 days with controlled parallelism
  • One UI for logs, misses, and ownership

Orchestration vs transformation

  • Orchestrator: when and in what order
  • Spark/dbt/SQL: how data is computed

Cron vs orchestrator

Cron can fire one script. It struggles with multi-step dependencies, partial failures, backfills, and visibility.

Interview tip: "Orchestration manages workflow; compute engines transform data." Name Airflow (or your stack) and give a 3-task dependency example.