Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. What is orchestration?

Pipelines & scenarios · Core Pipeline Concepts

What is orchestration?

Easypipe-13
orchestrationAirflowDAGschedulingdependencies

Question

What is pipeline orchestration, and what tools do data engineers use?

Solution

Orchestration is scheduling and coordinating pipeline tasks: order, dependencies, retries, concurrency, and alerts. The orchestrator usually does not do heavy compute; it starts and watches work elsewhere.

Airflow / Dagster / Prefect / Step Functions
        |
        |  triggers & watches
        v
  Spark · dbt · warehouse SQL · Python · API pulls

What orchestration gives you

  • Run B only after A succeeds
  • Retry A three times on failure
  • Backfill last 30 days with controlled parallelism
  • One UI for logs, SLA misses, and ownership

Orchestration vs transformation

  • Orchestrator: when and in what order
  • Spark/dbt/SQL: how data is computed

Cron vs orchestrator

Cron can fire one script. It struggles with multi-step dependencies, partial failures, backfills, and visibility.

Interview tip: "Orchestration manages workflow; compute engines transform data." Name Airflow (or your stack) and give a 3-task dependency example.

PreviousNext