Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Delta transaction log

File formats & storage · Table Formats in Depth

Delta transaction log

Mediumfile-formats-28
delta-laketransaction-logmetadatavacuumtime-travel

Question

What is in the Delta Lake _delta_log folder?

Solution

The _delta_log directory in a Delta Lake table contains an ordered series of commit files that record every transactional change made to the table since its creation. Each transaction produces a zero-padded JSON file containing atomic actions like adding files, removing files, updating metadata, or changing protocol versions. To keep reader initialization fast as commit counts grow, Delta periodically generates compacted Parquet checkpoint files that summarize table state so readers do not have to replay thousands of individual JSON commits.

Actions committed in delta log JSON files

_delta_log/                                                              ├── 00000000000000000000.json         (Commit 0: metadata, add files)    ├── 00000000000000000001.json         (Commit 1: add files, remove files)...                                                                      ├── 00000000000000000010.json         (Commit 10: add files)             ├── 00000000000000000010.checkpoint.parquet (Consolidated state of 0..10)└── _last_checkpoint                  (Points to latest checkpoint)      

Inside every JSON commit file, Delta stores discrete record actions:

  • add: Registers a newly written Parquet data file, including file path, byte size, modification time, and per-column min/max statistics for data skipping.
  • remove: Marks a previously active Parquet file as logically deleted (such as an old file rewritten during an UPDATE or OPTIMIZE).
  • metaData: Records changes to table schema, partition columns, or configuration properties.
  • protocol: Defines the minimum reader and writer versions required to read or write the table.
  • commitInfo: Logs execution provenance, such as the timestamp, operation type (MERGE, WRITE), user ID, and engine version.

Parquet checkpoint files

Without checkpoints, a reader inspecting a table with 50,000 commits would have to download and parse 50,000 JSON files across the network:

  • By default, Delta writes a Parquet checkpoint every 10 commits.
  • A new reader simply reads _last_checkpoint, loads the latest checkpoint Parquet file to get the current snapshot of active files, and replays only the few JSON files created after that checkpoint.
  • This keeps query planning time predictable regardless of table age.

Time travel queries and vacuuming

This log architecture powers time travel. A user can query VERSION AS OF 5 by having the engine replay commits up to version 5. When older versions are no longer needed, running VACUUM permanently deletes physical Parquet files marked with remove actions whose deletion timestamps exceed the configured retention window.

PreviousNext