Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Right to be forgotten in a data lake

Data quality · Governance, Privacy & Lineage

Right to be forgotten in a data lake

Harddata-quality-31
right-to-be-forgottengdpricebergdelta-lake

Question

A user asks to delete their data. How do you delete it from a data lake with immutable files?

Solution

Deleting a user record from an immutable data lake requires using open table formats that support row-level deletes, followed by snapshot expiration and physical file vacuuming to purge historical versions. For petabyte-scale lakes with vast cold storage footprints, crypto-shredding offers an alternative by deleting the user-specific encryption key.

Deleting rows from append-heavy object stores

Cloud object stores like S3 and GCS do not allow in-place file modifications, making single-row deletions challenging:

  • Open table formats: Apache Iceberg, Delta Lake, and Apache Hudi provide native delete commands. Depending on table configuration, they either rewrite Parquet data files without the requested user records or append deletion vectors that hide deleted rows during query reads.
  • Vacuuming and snapshot expiration: Running a delete statement creates a new metadata snapshot, leaving historical files intact for time travel. To achieve true physical erasure, platforms must run snapshot expiration and vacuum jobs that permanently remove orphaned Parquet files from cloud buckets after legal retention limits pass.
  • Raw zones and immutable backups: Raw ingestion buckets, landing files, and cold archives must also be scrubbed or partitioned by date so historical files containing deleted identifiers can be purged systematically.
User Deletion Request -> [Audit Registry]
                               |
         +---------------------+---------------------+
         |                                           |
 [Table Format Path]                         [Crypto-Shredding Path]
   1. Issue DELETE statement                   1. Lookup user KMS key ID
   2. Compact deletion vectors                 2. Destroy key in KMS vault
   3. Expire snapshots and VACUUM files        3. Ciphertext permanently unreadable

Operationalizing deletion requests requires end-to-end audit tracking:

  • Log every deletion ticket in a centralized compliance registry, recording user identifiers, request timestamps, and processing status.
  • Implement crypto-shredding for extreme scale: Encrypt sensitive user attributes using individual, per-user encryption keys managed in a key vault. When a deletion request arrives, destroying that specific key renders the user data permanently unreadable across all immutable files and backups without rewriting terabytes of data.

Verification of physical erasure

Regulatory audits require proving that deleted records cannot be recovered. Teams schedule automated verification queries that confirm deleted identifiers return zero rows across production tables and verify that old file objects no longer exist in object storage.

PreviousNext