Pipelines & scenarios · Design & Scenarios

Process 1TB efficiently

Hardperformance1TBParquetpartition pruneSparkcost

Interview question

How would you process 1TB of data efficiently in a pipeline?

How to answer

The short answer

1TB is routine at scale if you avoid full scans, tiny files, and unnecessary . Efficiency is about I/O, partitioning, and compute shape.

Output
Efficient path sketch
  columnar storage (Parquet/ORC) + compression
  partition by date (read only needed days)
  predicate pushdown / column prune
  scale workers horizontally
  minimize shuffle (filter early, broadcast small dims)
  avoid millions of small files (compact)

Concrete tactics

  1. 1Do not read all 1TB if the job only needs yesterday: prune.
  2. 2Use Parquet/ORC, not CSV, for analytics scans.
  3. 3Select only needed columns.
  4. 4Size files sensibly (often ~128MB–1GB range as a rule of thumb).
  5. 5Tune partitions/executors so tasks are even; fix skew.
  6. 6Prefer incremental processing over rewriting history daily.
  7. 7Cache only when reused multiple times in one job.

Cost mindset

Scan less data, shuffle less data, store compressed columnar. Cluster size is the last lever, not the first.

Interview tip: Start with "how much of the 1TB do I actually need to read?" Partition prune + columnar format is the fresher-to-strong answer.