Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Data lake folder layout

File formats & storage · Storage Layout

Data lake folder layout

Easyfile-formats-44
folder-layoutstorage-designbest-practiceshive-partitioningdata-lake

Question

How do you organize folders and paths in a data lake?

Solution

A data lake should be structured using a clean, hierarchical folder convention organized by zone, source system, table name, and partition (such as bronze/salesforce/accounts/ingest_date=2025-01-01/). Adopting Hive-style key=value directory naming ensures compatibility with query engines and catalog crawlers, while separating raw, immutable landing files from clean, transformed analytical data protects raw history from accidental modification. While modern table formats track physical file pointers through metadata logs and reduce reliance on folder layout, maintaining a standardized directory taxonomy remains essential for access control, billing attribution, and lifecycle policies.

Zone and source path conventions

s3://company-lakehouse-prod/                                              ├── landing/raw_salesforce/accounts/2025/01/01/dump.json  (Immutable, raw)├── bronze/salesforce/accounts/ingest_date=2025-01-01/data.parquet        ├── silver/crm/dim_accounts/country=US/part-0000.parquet                  └── gold/finance/fct_monthly_mrr/year=2025/month=01/part-0000.parquet     

Structuring folder hierarchies requires adhering to several fundamental conventions:

  • Zone separation: Divide buckets or root directories by architectural tier (landing, bronze, silver, gold). Store raw vendor payloads in an immutable landing bucket configured with append-only permissions and lifecycle auto-deletion.
  • Source and domain namespacing: Group tables by source system or business domain to simplify identity and access management boundary policies.
  • Naming hygiene: Use strictly lowercase alphanumeric characters, underscores, and hyphens. Never include spaces, URL-encoded characters, or special symbols like colons in S3 object keys because different query engine drivers handle string encoding inconsistently.

Hive-style partitioning semantics

Using Hive-style partition naming (ingest_date=2025-01-01/) allows tools like AWS Athena, Spark, and Glue crawlers to infer column names and partition values automatically during catalog discovery without custom regex parsing.

How modern table formats change path reliance

In modern table formats like Apache Iceberg and Delta Lake, the query engine no longer depends on directory structures to discover partitions or data files; the metadata log points directly to exact file URIs:

  • Iceberg supports object storage layout strategies that write files with randomized hash prefixes to distribute I/O evenly across S3 storage partitions.
  • Even so, human-readable directory structures remain best practice at the root level to support bucket-level cost monitoring and lifecycle tiering.
  • Keeping raw immutable files separated from processed tables guarantees that upstream data bugs can be replayed from source without corrupting analytical models.
PreviousNext