The short answer
1TB is routine at scale if you avoid full scans, tiny files, and unnecessary . Efficiency is about I/O, partitioning, and compute shape.
Efficient path sketch
columnar storage (Parquet/ORC) + compression
partition by date (read only needed days)
predicate pushdown / column prune
scale workers horizontally
minimize shuffle (filter early, broadcast small dims)
avoid millions of small files (compact)Concrete tactics
- 1Do not read all 1TB if the job only needs yesterday: prune.
- 2Use Parquet/ORC, not CSV, for analytics scans.
- 3Select only needed columns.
- 4Size files sensibly (often ~128MB–1GB range as a rule of thumb).
- 5Tune partitions/executors so tasks are even; fix skew.
- 6Prefer incremental processing over rewriting history daily.
- 7Cache only when reused multiple times in one job.
Cost mindset
Scan less data, shuffle less data, store compressed columnar. Cluster size is the last lever, not the first.
Interview tip: Start with "how much of the 1TB do I actually need to read?" Partition prune + columnar format is the fresher-to-strong answer.