Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Quarantine tables

Data quality · Building Quality In

Quarantine tables

Easydata-quality-27
quarantine-tablesdead-letter-queuepipeline-designerror-handling

Question

What is a quarantine (error) table, and how do you use it?

Solution

A quarantine table is an isolated storage destination where rows that fail quality checks are redirected while valid records continue flowing downstream. This pattern prevents a single malformed row from halting an entire batch job while capturing diagnostic metadata needed to fix and reprocess rejected data.

Isolating malformed records with dead letters

In high-throughput pipelines, dropping an entire batch because two rows have invalid postal codes creates unacceptable delays for business users. Quarantine tables solve this through conditional branching:

  • Valid records continue: Rows passing all schema, type, and business constraints are written directly to production target tables.
  • Invalid records divert: Rows failing validation are written to a quarantine table alongside operational metadata, including the pipeline run identifier, rejection timestamp, the specific rule that failed, and the raw source payload.
Incoming Batch -> [Validation Filter]
                        |
       +----------------+----------------+
       |                                 |
 [Valid Rows]                    [Invalid Rows]
       |                                 |
       v                                 v
Target Marts / Silver           Quarantine Table (run_id, error_reason, raw_record)

Operating a quarantine pattern requires disciplined maintenance practices:

  • Monitor quarantine volume closely. Diverting records keeps the main pipeline running, but a sudden spike in quarantined rows indicates an upstream systemic failure, such as a changed API contract or dropped database column.
  • Never let quarantine tables hide systemic failures. If the quarantine rate exceeds a defined threshold, such as one percent of the batch, configure the pipeline to alert on-call engineers.
  • Reprocess rejected records once bugs are resolved. After the upstream schema or transformation logic is fixed, run replay scripts that re-evaluate quarantined rows and insert the corrected data back into production.

Operational governance of error tables

Without automated monitoring, quarantine tables turn into silent data graveyards. Establishing weekly review cadences and setting expiration retention policies keeps the quarantine backlog clean and accountable.

PreviousNext