Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. OpenLineage

Data quality · Governance, Privacy & Lineage

OpenLineage

Mediumdata-quality-34
openlineagemetadatadata-lineageairflow

Question

What is OpenLineage?

Solution

OpenLineage is an open standard and framework designed to capture operational data lineage metadata from pipeline engines as jobs execute. It provides a vendor-neutral JSON specification that allows different processing frameworks to report job runs, dataset inputs, and outputs into a centralized lineage repository.

Standardized runtime metadata telemetry

Before OpenLineage, every pipeline framework used proprietary metadata formats, making it nearly impossible to trace data moving across multiple technologies. OpenLineage solves this fragmentation:

  • Core data model: Defines explicit specifications for jobs as named processes, runs as individual executions with start and complete states, and datasets as tables, topics, or object store paths consumed as inputs or created as outputs.
  • Runtime event emission: Integrates directly into execution runtimes via plugins or listeners. As a pipeline runs, it emits OpenLineage events over HTTP or Kafka without requiring changes to business transformation logic.
  • Supported processing engines: Integrates with Apache Airflow to capture DAG task dependencies, Apache Spark via query plan listeners, dbt via model execution hooks, and Apache Flink for real-time streaming state.
[Airflow DAG] --+
[Spark Batch] --+--> OpenLineage Events (JSON) --> [Marquez / Central Catalog]
[dbt Run]     --+                                              |
[Flink Stream]--+                                  Unified Global Lineage Graph

Emitted metadata events flow into centralized visualization and governance tools:

  • Marquez serves as the open-source reference implementation, storing lineage events and providing an interactive graph UI with full run history.
  • Commercial data catalogs and observability platforms ingest OpenLineage events to build enterprise-wide dependency graphs.
  • Enables cross-tool lineage: An engineer can trace an event from a Kafka topic ingested by Flink, processed by Spark, orchestrated by Airflow, and modeled in dbt within a single continuous graph.

Operational benefits of open lineage standards

Adopting an open lineage standard eliminates vendor lock-in and prevents blind spots across hybrid architectures. Teams gain immediate root-cause analysis during pipeline incidents and reliable impact assessments before deploying schema changes.

PreviousNext