Data Engineering Glossary

The structural blueprint defining the tables, columns, data types, and integrity constraints of a database.

A schema tells the engine what columns exist, whether missing values are allowed, and how tables link together using keys.

Writing code that specifies what data you want rather than the low-level physical steps to fetch it.

SQL is declarative: you describe your needed output with SELECT and WHERE, leaving the physical execution strategy to the query optimizer.

A column in one table that references the primary key of another table to maintain relationships.

Foreign keys enforce referential integrity so you cannot create an order referencing a customer_id that does not exist in the customers table.

The fixed sequence in which a database engine evaluates SQL clauses: FROM, WHERE, GROUP BY, HAVING, SELECT, DISTINCT, ORDER BY, LIMIT.

Even though SELECT is written first, the database executes FROM and WHERE first to locate and filter records before evaluating columns and aliases.

An identifier that already exists in the real-world operational source system, such as a customer email or order UUID.

Natural keys originate in transactional apps. In the warehouse, dimensions map natural keys to surrogate keys to manage multiple historical versions cleanly.

A special SQL marker indicating that a data value is missing, unknown, or not recorded.

NULL is not zero, false, or an empty string. Because it represents missing data, any equality comparison with NULL evaluates to UNKNOWN.

A condition in a query (such as WHERE status = 'delivered') that evaluates to TRUE, FALSE, or NULL for each row.

Predicates determine which rows pass through a filter and reach subsequent aggregation or join stages.

A column or set of columns that uniquely identifies each individual row in a table.

A primary key prevents duplicate records and cannot contain NULL values. In ShopKart, order_id is the primary key of the orders table.

The database engine component that calculates the fastest physical execution strategy for a SQL query.

The optimizer inspects table statistics, row counts, and indexes on disk to choose the most efficient join and filtering plan.

A structured grid of data organized into named columns with declared types and horizontal rows.

Unlike an unstructured file, a relational table enforces schemas and data types. Each row represents a single entity or event, such as an order or customer.

An artificial numeric or hashed identifier generated specifically for database tables instead of using business keys.

Surrogate keys isolate warehouse dimensions from operational database key reassignments and enable tracking historical changes across multiple versions of the same entity.

The SQL logic system where every comparison evaluates to TRUE, FALSE, or UNKNOWN because of NULL.

Because NULL represents missing information, expressions like `column = NULL` evaluate to UNKNOWN rather than TRUE or FALSE. Use `IS NULL` or `IS NOT NULL` instead.

The number of distinct or unique values contained within a specific column or dataset.

High-cardinality columns (like user_id) have mostly unique values, while low-cardinality columns (like payment_status) have only a few distinct values. Cardinality dictates how query engines plan filters and group operations.

An unintended multiplication of row counts when joining on a key that is not unique on both sides.

If an order joins against a payments table with multiple transaction attempts per order on a non-unique key, order totals get duplicated and sum to false amounts.

Organizing relational database tables to reduce data redundancy and prevent update anomalies.

Instead of repeating a customer's full address in every order row, normalisation stores the address once in a customers table and links orders via customer_id.

Organizing relational database tables to reduce data redundancy and prevent update anomalies.

Instead of repeating a customer's full address in every order row, normalization stores the address once in a customers table and links orders via customer_id.

An operation that combines columns from two or more tables based on a related column between them.

Joins connect normalized relational tables using primary and foreign key predicates, allowing queries to reconstruct complete business entities.

The specific subset of rows around the current row over which a window function calculates its result.

Frames like `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` define running totals, while moving averages use bounded offsets like `ROWS BETWEEN 6 PRECEDING AND CURRENT ROW`.

A function that calculates an aggregate or ranking across a group of related rows without collapsing them into a single row.

Window functions use the OVER clause with PARTITION BY and ORDER BY to compute running totals, rankings, and lead/lag values while preserving every input row.

A named temporary result set defined with the WITH clause that makes complex queries readable and modular.

CTEs break down long pipelines into sequential named steps instead of deeply nested subqueries, making data flow explicit and easier to debug.

A nested query that references columns from the outer query, evaluating once per outer row.

Unlike uncorrelated subqueries that run once upfront, correlated subqueries behave like nested loops. In batch pipelines, they often cause severe performance bottlenecks unless rewritten as joins or window functions.

A CTE that references itself to iteratively traverse hierarchical or graph-structured datasets.

Recursive CTEs combine an initial anchor query with a recursive member using UNION ALL, repeatedly querying child rows until no further records match.

SQL operators that combine rows from two or more queries vertically into a single result set.

UNION ALL appends rows without deduplication, INTERSECT returns matching rows between queries, and EXCEPT produces rows present in the first set but absent from the second.

A SQL query nested inside an outer query to compute intermediate values or filter results.

Subqueries can return a single scalar value, a list of values for IN filters, or an entire table in the FROM clause, executing before or alongside the outer statement.

A query predicate written so that the database engine can utilize an index instead of scanning every table row.

Wrapping a column in a function (such as `WHERE YEAR(created_at) = 2026`) prevents index lookups. Writing `WHERE created_at >= '2026-01-01' AND created_at < '2027-01-01'` is SARGable and fast.

A mathematical way to describe how the execution time or memory footprint of code scales as data grows.

O(1) means the work stays the same however big the data is. O(N) means the work grows in step with the number of rows. O(N squared) means doubling the data makes the work 4 times bigger, which slows pipelines to a crawl.

The classification of a value (such as int, float, str, or bool) determining what operations can be performed on it.

Data types prevent invalid operations. In data pipelines, raw CSV inputs arrive as text strings and must be converted into numerical types before doing math.

A quarantine storage location where corrupted or invalid records are isolated without halting the main pipeline.

Instead of crashing an entire batch when a single row has bad data, defensive pipelines route failed rows to a dead-letter queue for inspection.

A collection of key-value pairs backed by an in-memory hash table for fast lookups.

Dictionaries find a value straight from its key, however big the dict is. They are widely used in data pipelines for row representations, dimension lookups, and JSON parsing.

A Python function that yields items lazily on demand rather than storing the complete result list in memory.

Generators use the `yield` keyword to stream records from large CSV or JSON files with constant, tiny memory usage.

An unordered collection of unique values backed by an in-memory hash table.

Sets do not allow duplicate items and can answer 'is this value here?' very quickly. They are ideal for deduplicating IDs and filtering out blacklisted keys.

A Python object capable of returning its members one at a time, such as a list, tuple, or generator.

In data pipelines, iterables let you process massive datasets row by row or chunk by chunk without loading an entire multi-gigabyte file into RAM simultaneously.

An ordered, mutable sequence of values enclosed in square brackets [1, 2, 3].

Lists keep their order and you can add, remove or change items. Checking whether a value is in a list makes Python look at the items one by one, so it gets slower as the list grows.

An ordered, immutable sequence of values enclosed in parentheses (1, 2).

Tuples cannot be altered once created. Because they are immutable, they take less memory than lists and can be used as dictionary keys.

Explicitly converting a value from one data type to another, such as turning the text string '49.50' into a float.

Pipelines read CSV files where every field starts as a string. Type casting converts those strings into integers or floats so calculations run accurately.

A named label that points to a specific value stored in computer memory.

When you write price = 500, Python allocates an integer object in RAM and attaches the label price to it. Reassigning moves the label to a new object.

A Python construct used with the 'with' statement that guarantees resource allocation and cleanup.

A context manager defines __enter__ and __exit__ methods. It guarantees that files, database connections, and network sockets are closed cleanly even if an unhandled error occurs.

A dynamic value set in the operating system environment outside of application code.

Pipelines read environment variables (via os.environ) for secrets, tokens, and database passwords, keeping sensitive credentials out of version control.

An event or signal raised during program execution that disrupts normal instruction flow when an unexpected condition occurs.

Python raises built-in exceptions like ValueError, KeyError, and FileNotFoundError. Data pipelines handle them defensively using try and except blocks to avoid unhandled crashes.

A retry strategy that doubles the waiting duration between successive failed attempts.

When an external API or database is overloaded, exponential backoff (e.g. 1s, 2s, 4s, 8s) gives the struggling service time to recover rather than flooding it with immediate retries.

A 3-digit numerical code returned by a web server indicating the outcome of an HTTP request.

Status codes inform the client whether a request succeeded (200 OK), was client-invalid (400, 401, 404), was rate-limited (429), or failed on the server (500, 503). Pipelines retry 429 and 5xx errors while aborting on 4xx errors.

The property where running an operation multiple times produces the exact same result as running it once.

A pipeline is idempotent if rerunning it after a network crash does not create duplicate rows or alter correct metrics. Every reliable production data pipeline must be idempotent.

Random variance added to retry delay intervals to prevent concurrent clients from retrying at identical moments.

Adding random jitter to exponential backoff desynchronizes parallel worker retries, preventing the Thundering Herd Problem where hundreds of tasks hammer an API simultaneously.

JavaScript Object Notation: a standard text-based format for representing structured data using key-value pairs and arrays.

JSON is the universal payload format for REST APIs and web services. Python converts JSON to and from native dictionaries using json.loads and json.dumps.

A server-side policy restricting the number of requests a client can make within a specified time window.

When a data pipeline sends requests too rapidly, servers return HTTP 429 Too Many Requests. Pipelines manage rate limits using pacing, batching, and backoff delays.

An HTTP-based interface allowing programs to exchange structured data over a network using standard request methods.

Representational State Transfer (REST) APIs let data pipelines fetch data from external services over HTTP using GET and POST methods, typically returning JSON payloads.

A report showing the sequence of function calls that led to an unhandled exception or error in Python.

When Python halts on an error, the traceback prints line numbers and file names from bottom to top, pointing directly to the offending line of code.

Explicit annotations in Python code indicating the expected data types of function arguments and return values.

Type hints (`def process_order(order_id: int) -> dict:`) catch subtle type errors before code runs in production and make pipeline contracts obvious to teammates.

Filtering rows in a DataFrame or array using an array of True/False boolean conditions.

In Pandas, boolean expressions like df['amount'] > 100 produce a boolean mask used to select only matching rows without writing explicit Python loops.

A two-dimensional tabular data structure with labelled rows and columns held in memory.

In Pandas and PySpark, DataFrames represent structured tables in memory. Each column has a distinct data type, supporting relational operations like filtering, grouping, and joining.

A one-dimensional labelled array holding data of a single type in Pandas.

A Pandas DataFrame is composed of multiple Series objects sharing a common index, with each Series representing a single column.

Executing operations across an entire array of values simultaneously in compiled C code rather than slow Python loops.

Libraries like NumPy and pandas use vectorized operations to apply transformations to millions of rows orders of magnitude faster than iterating with Python for-loops.

A Python class decorator (@dataclass) that automatically generates boilerplate methods for storing typed data attributes.

Dataclasses provide clear data contracts and type safety in pipeline records, replacing untyped dictionaries with structured objects having explicit fields.

Emitting log events as structured key-value pairs (usually JSON) rather than plain unstructured text strings.

Structured logging allows monitoring systems like Datadog, CloudWatch, and Elasticsearch to parse, filter, and alert on pipeline log fields such as row counts, latency, and status codes.

An automated test that validates an isolated piece of program logic (such as a single function) against expected inputs and outputs.

Data engineers write unit tests using frameworks like pytest to verify data transformation functions, edge cases, and cleaning rules before code is deployed.

Processing bounded groups of data collected over a window of time at scheduled intervals.

Batch jobs run on a schedule, such as hourly or nightly, to extract, clean, and aggregate accumulated records in high-throughput chunked workloads.

The raw, immutable landing zone in a medallion lakehouse architecture.

The Bronze layer preserves incoming data byte-for-byte in append-only storage. No transformations or business logic are applied so raw history can always be audited or replayed.

A set of automatic steps that moves data from where it is created to where people use it, cleaning it on the way.

A pipeline has sources (where data starts), steps in the middle (collect, store, clean) and a destination (tables or dashboards). Data engineers build pipelines and keep them running.

A data pattern where raw data is loaded directly into storage and transformed using the warehouse's own query engine.

ELT leverages modern distributed warehouse engines (Snowflake, BigQuery, DuckDB) to transform raw data in place using SQL models, keeping raw lineage intact.

A data pipeline pattern where data is extracted from sources, transformed in a separate compute engine, and loaded into storage.

ETL transforms data outside the warehouse before storage. It was common when warehouse storage was scarce; today ELT is more common for batch reporting.

The step that copies data from a source system into your own storage.

Ingestion only collects data. It should not change or throw away anything, so that you always keep an exact copy of what the source sent.

How long it takes for data to travel from where it is created to where it can be used.

A nightly job has a latency of many hours. A streaming job can have a latency of a few seconds. Lower latency usually costs more to build and run.

Analytical databases optimized for reading and summarizing millions of rows across columnar tables.

OLAP systems like Snowflake, BigQuery, or DuckDB scan columns instead of rows to run aggregates like SUM and AVERAGE orders of magnitude faster than OLTP systems.

Operational databases optimized for fast row-level insertions, updates, and lookups.

OLTP databases like PostgreSQL or MySQL power customer-facing apps. They prioritize ACID guarantees, low-latency writes, and high concurrency over analytical reporting.

Information that can identify a person, such as a name, phone number, email or card number.

Laws and company rules often say personal data must be hidden or protected before many people can see it. This is one reason a team may clean data before loading it.

A promise about how fresh or how reliable the data will be, for example 'the report is ready by 7 AM'.

An SLA turns a vague wish into a number you can measure. It decides how fast a pipeline must be and how much effort it is worth.

Processing unbounded real-time event streams continuously as individual messages arrive.

Streaming platforms like Kafka or Kinesis process records one by one or in micro-batches with low latency, providing immediate updates for dashboards and alerts.

A table providing descriptive context (who, what, where, when) for numeric measurements in fact tables.

Dimension tables hold business attributes like customer city, product category, and campaign source used to slice and filter fact measurements in BI tools.

A table in a dimensional model containing measurable numeric business events and foreign keys to dimension tables.

Fact tables record business events that happen at a specific point in time, such as order_amount, discount_value, and quantity_ordered in fact_sales.

The exact level of detail represented by a single row in a fact or dimension table.

Defining grain is the foundational first step of data modelling. For example: '1 row per line item in a customer order' versus '1 row per completed order summary'.

A dimensional schema where a central fact table references multiple surrounding denormalized dimension tables.

Star schemas simplify business reporting queries, reduce join complexity compared to deeply normalized models, and enable columnar query engines to scan data efficiently.

A database design standard where every non-key column depends solely on the primary key, eliminating transitive dependencies.

3NF is optimal for write-heavy transactional systems (OLTP) to prevent inconsistent updates, whereas star schemas denormalize for fast analytical reads (OLAP).

An update strategy that overwrites old dimension attributes with current values, maintaining no history.

If customer 42 moves from Bengaluru to Mumbai, SCD Type 1 replaces 'Bengaluru' with 'Mumbai'. Past sales now show under Mumbai, which simplifies reporting when history is unneeded.

An update strategy that inserts a new row with valid date ranges whenever an attribute changes, preserving full history.

SCD Type 2 stores both the past Bengaluru address and current Mumbai address using `valid_from`, `valid_to`, and `is_current` flags so historical sales reports remain accurate.

An open-source, columnar storage file format optimized for fast analytical queries and efficient compression.

Parquet stores column statistics (min/max values) inside file headers so query engines can skip entire blocks without reading them from disk.

A file format layout where values from the same column are stored together contiguously on disk rather than row by row.

Columnar storage allows queries calculating `SUM(amount)` to read only the bytes for that single column, bypassing all other columns completely and compressing data tightly.

An optimization where a query engine skips reading entire folders or files because partition filter criteria do not match.

If data is partitioned by `year=2026/month=09`, a query for September 2026 never reads files from October or August, cutting scan volume and cloud egress costs dramatically.

Evaluating query filter conditions directly at the storage level to skip reading non-matching data blocks.

Columnar formats like Parquet store min/max statistics in file footers. If a filter requires amount > 1000 and a row group has max_amount = 450, the engine skips reading that entire block.

A performance degradation caused by millions of tiny files overwhelming object storage metadata and network connections.

Frequent streaming writes or over-partitioning generate thousands of tiny files. Query engines spend excessive time negotiating HTTP requests rather than reading data, requiring scheduled compaction jobs.

A data lakehouse design pattern that organizes data into bronze (raw), silver (cleaned), and gold (business star schema) layers.

Bronze preserves raw immutable events, silver deduplicates and casts types, and gold builds tested aggregations for dashboards and business analysts.

A cloud security framework that defines which users, services, and pipelines have permission to access specific resources.

IAM policies follow the principle of least privilege: an ingestion pipeline gets write-only access to the bronze bucket, never full admin privileges.

Scalable cloud storage (like AWS S3 or Google Cloud Storage) that manages data as immutable objects rather than file trees.

Object storage provides virtually unlimited capacity and high durability at low cost, serving as the foundational storage layer for modern data lakes.

A centralized metadata registry (such as AWS Glue Data Catalog) storing table schemas, partition locations, and data types.

A data catalog lets SQL engines discover and query files in object storage without hardcoding physical S3 file paths inside queries.

Applying a structure or schema to raw data at query execution time rather than when loading into storage.

Engines like Amazon Athena or DuckDB project table structures over raw Parquet or JSON files on S3 when a query runs, avoiding heavy pre-loading ETL steps.

A warehouse architecture where multiple independent compute nodes process portions of a query simultaneously across shared or distributed storage.

MPP warehouses (like Redshift, BigQuery, and Snowflake) distribute table data across slices or workers so massive scans finish in seconds rather than hours.

A collection of connected server computers working together under a coordinator to execute workloads.

A Spark cluster consists of a coordinator node (Driver), worker nodes (Executors), and a cluster manager (such as Kubernetes or YARN).

Software that manages and allocates compute resources (CPU cores and RAM) across cluster nodes.

The cluster manager (such as Kubernetes or YARN) allocates container slots where Spark executors launch and run tasks.

The dependency graph recording the exact sequence of transformations used to construct each partition.

Lineage provides in-memory fault tolerance: if an executor crashes and loses a partition, Spark recomputes only the lost partition from source.

Coordinating multiple networked computers to process and store datasets that exceed the capacity of a single machine.

Rather than buying an expensive single computer with terabytes of RAM, distributed computing connects dozens of commodity machines working in parallel.

Postponing computation until an action (like count, write, or collect) explicitly requires a result.

In PySpark, transformations like `.filter()` and `.select()` do not execute immediately; Spark builds an execution graph and optimizes it before running.

An operation where each output partition depends only on a single input partition without network data movement.

Operations like `select()` and `filter()` run independently on each executor without exchanging data across the cluster network.

A discrete chunk of dataset rows that can be processed independently by a single CPU core.

Spark splits large files into partitions (typically up to 128 MB each) so executor cores can process different slices of data simultaneously.

The physical redistribution of data across cluster network nodes during wide transformations.

Shuffling writes intermediate data to disk and transmits it across the network. Minimizing unnecessary shuffles is key to tuning Spark performance.

The central coordinator process in a Spark cluster that plans execution, builds the DAG, and assigns tasks to executors.

The driver converts your PySpark code into physical tasks. Collecting millions of rows back to the driver with `.collect()` is a frequent cause of driver Out-Of-Memory crashes.

Worker processes on cluster nodes that execute assigned compute tasks and store partition data in memory or disk.

Executors perform the actual filtering, joining, and aggregation work in parallel across distributed cluster nodes.

The unified entry point for programming Spark with the DataFrame API.

A SparkSession encapsulates the cluster connection, configurations, catalog metadata, and SQL execution runtime.

A group of tasks in a Spark job that can execute in parallel without transferring data across the network.

Narrow transformations stay within one stage. Wide transformations force a shuffle boundary that starts a new stage.

The smallest unit of execution in Spark, representing work sent to one CPU core for 1 partition.

Within an execution stage, Spark creates exactly one task for every data partition. Cores execute tasks in waves until the stage finishes.

An operation that requires data from multiple input partitions to be grouped, sorted, and redistributed across executors.

Operations like `groupBy()`, `distinct()`, and joins require a network shuffle to align matching keys onto the same executor nodes.

Spark's query optimizer that analyzes, optimizes, and compiles DataFrame operations into efficient physical execution plans.

Catalyst takes declarative DataFrame operations and applies relational rules like predicate pushdown, column pruning, and whole-stage code generation before execution.

An optimizer optimization where the query engine scans only the specific columns needed, skipping all other columns on storage.

In columnar formats like Parquet, column pruning skips bytes for unreferenced columns directly on cloud storage, saving substantial network I/O and memory.

The detailed breakdown produced by df.explain() showing the parsed, analyzed, optimized, and physical execution stages.

Examining an explain plan allows data engineers to confirm whether predicate pushdown occurred and where network shuffles (Exchange operators) take place.

Spark's physical execution engine that manages off-heap binary memory and generates compiled Java bytecode.

Tungsten bypasses JVM garbage collection overhead and optimizes CPU cache locality by packing records directly into off-heap binary format.

The PySpark schema object that defines a table's column names, data types, and nullability constraints explicitly.

Declaring an explicit StructType prevents expensive schema-inference passes across cloud storage and guards against unexpected upstream type drift.

Custom Python code registered to transform DataFrame column values when built-in functions cannot express the logic.

Standard Python UDFs suffer severe serialization overhead passing data between the JVM and Python worker processes. Built-in functions should always be preferred.

An optimized join where Spark copies a small lookup table to every executor, eliminating network shuffles of the large table.

Broadcasting a small dimension table (e.g. 50 MB category lookup) avoids shuffling a multi-gigabyte fact table across the cluster.

A narrow transformation that decreases partition count by merging adjacent partitions locally without a full network shuffle.

coalesce() avoids shuffle overhead when reducing partition counts before writing to storage, though it cannot increase partition count.

An uneven distribution of records across partitions where a single worker receives vastly more data than others.

If 80% of orders come from a default customer_id, one executor runs for hours while others finish in seconds. Techniques like salting fix skew.

A wide transformation that redistributes data across the cluster into a new number of partitions using a full network shuffle.

repartition() creates roughly equal-sized partitions via round-robin or hash partitioning, useful for increasing parallelism or eliminating severe partition imbalance.

Appending a random integer prefix to hot join keys to spread skewed records evenly across multiple worker partitions.

Salting divides a hot key into N sub-keys, distributes the join load across N executors, and avoids single-executor bottlenecks.

A distributed join where data is shuffled by key, and one side builds an in-memory hash table joined against the streaming second side.

Used when one side is moderately small per partition and keys do not need sorting, saving sorting overhead compared to Sort Merge Join.

The default distributed join in Spark: both sides shuffle by key, sort partition records, and merge them in a linear pass.

SMJ scales to massive datasets because it does not require fitting either side into memory, but it incurs both shuffle network transfer and disk sorting costs.

Spark feature that dynamically re-optimizes query physical plans at runtime using actual intermediate stage statistics.

AQE can automatically coalesce small shuffle partitions, switch Sort Merge Joins to Broadcast joins if data shrinks, and split skewed partitions at runtime.

Writing in-memory executor partition data to local disk when available execution or storage RAM is exhausted.

Spilling prevents Out-Of-Memory crashes, but degrades pipeline performance by 5x to 20x due to disk I/O and deserialization penalties.

A Python class defining a template for a single atomic task, such as executing a Python function, running a Bash command, or submitting a Spark job.

Operators represent the 'what' in a pipeline. Airflow provides native operators (PythonOperator, BashOperator) and provider operators (EmrAddStepsOperator, BigQueryInsertJobOperator).

An Airflow DAG setting that controls whether the scheduler runs all missed historical intervals between start_date and the current date when unpaused.

Setting catchup=False prevents an accidental flood of dozens or hundreds of backfill runs when enabling a new DAG for the first time in production.

A mathematical graph of tasks connected in one forward direction without loops, defining execution order.

Orchestration engines like Apache Airflow model pipelines as DAGs so tasks run only after upstream dependencies succeed.

The specific data interval timestamp that an orchestrated pipeline run is intended to process, distinct from wall-clock execution time.

A pipeline running at 03:00 on Tuesday morning has a logical date covering Monday's data (2026-10-05), guaranteeing predictable idempotency during backfills.

A specific run of a task for a particular logical date and DAG run, with its own lifecycle state (queued, running, success, failed).

While a task is declared once in Python code, a Task Instance represents that task executing for a specific interval, such as the 2026-09-06 daily batch.

The modern Airflow 2 syntax using Python decorators (@dag and @task) to define pipelines and pass data between tasks automatically.

Instead of declaring separate Operator classes and manual xcom_push calls, the TaskFlow API lets Python functions exchange return values directly as XCom arguments.

A mechanism in Apache Airflow allowing tasks within the same DAG run to exchange small metadata payloads, such as an S3 URI or batch ID.

XCom serializes values into the Airflow metadata database. Production pipelines use XCom only for small strings and metadata pointers (< 48 KB), never for large DataFrames or files.

A concurrency control mechanism in Airflow that limits how many simultaneous task instances can access a shared resource.

Setting an Airflow pool with 5 slots around a database extract task ensures that backfills with 365 DAG runs never overwhelm the operational database with concurrent queries.

A specialized operator that pauses execution until an external criteria is met, such as an S3 key appearing or a partition landing.

Sensors should use mode='reschedule' or deferrable triggers so they release worker slots while waiting, avoiding worker slot starvation.

Running an orchestration pipeline over historical time intervals to process past data or replay logic changes.

When schema logic or a bug fix is deployed, an idempotent pipeline can be backfilled over the past 30 days without corrupting current figures.

An Airflow operator that suspends itself and yields its worker thread to an asynchronous Triggerer process while waiting for an external task to complete.

Deferrable operators prevent long-running external compute jobs (like 2-hour EMR or Databricks runs) from holding scarce Airflow worker slots idle.

An Airflow feature (.expand()) that generates a variable number of task instances at runtime based on upstream output.

Instead of hardcoding 10 tasks in code, dynamic task mapping accepts a list of partitions or files from an upstream task and creates one parallel task per item on the fly.

A setting that controls the condition under which an Airflow task is allowed to run based on upstream task states.

The default trigger rule is all_success, meaning every parent task must succeed. Other options like none_failed or all_done allow branches and cleanup tasks to execute even when earlier tasks fail or get skipped.

A task instance that the metadata database still marks as running, but whose operating system worker process has vanished or died unexpectedly.

Zombies commonly occur when a worker container gets killed by the host kernel due to memory exhaustion (SIGKILL / OOM). The Airflow scheduler detects zombies during its periodic heartbeat sweep and marks them failed.

A strategy defining how dbt persists a model in the database: view, table, incremental, or ephemeral CTE.

Staging models are typically materialized as views to keep queries lightweight without duplicating raw data. High-volume dimensional models are materialized as tables or incremental models to minimize query compute for BI analysts.

A dbt macro that references another model by name and automatically builds the dependency DAG and target table names.

Using `{{ ref('stg_orders') }}` tells dbt to build the staging table first and replaces the text with the correct environment schema path automatically.

A dbt feature that automatically captures point-in-time state changes on mutable tables using Slowly Changing Dimension Type 2 (SCD-2) rules.

dbt Snapshots monitor source tables and automatically maintain valid_from and valid_to timestamps when records update, preserving historical accuracy.

A dbt macro that points to a raw, untransformed table loaded from an external producer into the warehouse.

Declaring sources in YAML enables dbt to test source freshness, track data lineage from the landing edge, and document table owners.

The templating engine used by dbt to write dynamic SQL with variables, loops, conditionals, and macros.

Jinja evaluates expressions enclosed in curly braces like {{ ref('model_name') }} before compiling plain SQL that runs directly inside the warehouse.

An automated pipeline safeguard that halts downstream tasks immediately when severe data anomalies or corruption exceed tolerance thresholds.

If an ingestion run produces 0 records when 100,000 were expected, or if gross merchandise value drops by 80% due to an API outage, a circuit breaker halts execution immediately to prevent publishing corrupt metrics to executives.

A formal agreement between upstream application teams and data platform teams defining schema structure, field types, nullability, SLA freshness, and allowed value constraints.

Data contracts prevent software engineers from deploying silent transactional schema changes (like renaming a column or adding unexpected nulls) that would break downstream analytical models and dashboards.

A reusable data assertion defined in schema YAML, such as unique, not_null, accepted_values, or relationships.

Generic tests compile into SQL queries that look for failing rows. If the query returns zero rows, the test passes; if any bad rows return, the test fails.

A custom SQL query stored in the tests/ directory that asserts domain-specific business invariants on your models.

Like generic tests, singular tests return rows that violate business rules (such as order discounts exceeding total price). Zero returned rows signifies success.

An isolated side table where corrupt or non-compliant records are routed for asynchronous triage without terminating the entire batch pipeline.

When 15 out of 1,000,000 rows fail validation (such as a negative price or invalid postal code), the quarantine router writes the bad rows to a quarantine table for investigation while allowing clean rows to reach analytics.

A recorded timestamp or monotonic sequence number marking the latest record successfully ingested by an incremental pipeline.

The next pipeline run queries `WHERE updated_at > last_high_watermark` with a small lookback window to fetch only newly modified records.

A warehouse transformation model that processes strictly newly created or modified records since the prior high watermark instead of rebuilding the entire dataset.

Incremental models use a unique key and merge strategy to upsert changed records into existing warehouse tables, dropping warehouse compute from O(N) to O(ΔN).

Records that arrive at ingestion hours or days after their actual event occurrence due to mobile offline caching or network delays.

Incremental pipelines use lookback windows (e.g. querying the last 2 hours instead of 5 minutes) and upserts to incorporate delayed events safely.

A containerization platform that packages application code alongside its OS dependencies, runtime, and libraries into an isolated, reproducible container image.

Data engineers use Docker to run local development environments matching production, ensuring PostgreSQL, Airflow, Spark, and MinIO run identically across all developer laptops without host dependency conflicts.

A tool for defining and running multi-container Docker applications using a single YAML configuration file.

Docker Compose coordinates multiple interconnected services (such as PostgreSQL database, MinIO object store, and Apache Airflow) with unified networking, volume mounts, and environment configuration through a single command.

A database pagination technique that uses indexed column comparisons (WHERE id > last_seen_id) to seek directly to records without scanning offset rows.

Unlike OFFSET pagination which scans and discards N prior rows causing high database CPU and replica lag, keyset pagination seeks in O(log N) time using B-tree indexes, making large-scale database ingestion fast and predictable.

The curated, aggregated business reporting tier of the medallion architecture, optimized for analytics and BI dashboards.

The gold tier organizes cleansed silver tables into dimensional star schemas (fact and dimension tables) or denormalized wide aggregate rollups ready for executive reporting and low-latency queries.

The cleansed, validated, and conformed second tier of the medallion architecture.

In the silver tier, raw bronze data is parsed, deduplicated, enriched with type constraints, and cleansed. Null checks and business rules are enforced before dimensional modeling.

A coordinated set of consumers that divide topic partitions among themselves to process streaming events in parallel.

If a topic has 6 partitions and a consumer group has 3 instances, each worker handles 2 partitions, scaling stream processing throughput horizontally.

A sequential integer assigned to each message within a topic partition that uniquely identifies its position in the commit log.

Consumers maintain their current offset so that if a consumer worker crashes, it resumes reading from the exact next unprocessed message upon restart.

A category or feed name in a message broker to which producer services publish ordered records.

In ShopKart, user click events publish to `events.checkout.clicks` while payment events publish to `events.payments.transactions`.

A message delivery guarantee where no events are lost, but network retries may occasionally produce duplicate records.

Because network timeouts can happen after a consumer processed a message but before committing its offset, downstream sinks must be idempotent (using upserts or deduplication).

Capturing insert, update, and delete events directly from a database's write-ahead transaction log and streaming them to consumers.

Instead of running heavy SQL queries like `SELECT * FROM orders` against production transactional databases every night, CDC tools (like Debezium) stream change events continuously without table locking.

An open-source distributed Change Data Capture (CDC) platform built on top of Apache Kafka Connect that streams row-level changes from database transaction logs.

Debezium tails operational database WALs (such as PostgreSQL pgoutput or MySQL binlog) and emits JSON or Avro event envelopes containing before and after states for every row mutation.

An append-only log on disk where database engines record every transaction change before committing it to data tables.

The Write-Ahead Log guarantees database durability and crash recovery. Change Data Capture tools like Debezium read the WAL directly to stream database insertions, updates, and deletions into Kafka without issuing SQL queries.

Continuous Integration and Continuous Delivery: automated workflows that test, lint, build, and deploy code changes on every pull request.

In data engineering, CI/CD pipelines run automated SQLFluff linters, unit tests, and dbt test validations in ephemeral environments, preventing broken models or syntax errors from reaching production catalogs.

Unexpected modifications to source data structure, such as renamed columns, new fields, dropped attributes, or altered data types.

When upstream backend microservices change a database column or emit null values without notice, schema drift breaks downstream analytical DAGs unless automated schema contracts or defensive evolution policies are in place.

A centralized technical catalog describing table schemas, column data types, allowed values, foreign key relationships, and business grain.

A comprehensive data dictionary tells data analysts and downstream consumers exactly what each column means, its primary key constraints, and how timestamps are formatted, preventing conflicting business interpretations.