Abhishek Chauhan

Pro

@abhishekchauhan

Pro member

SRE-turned-DE. Care about observability, cost, and GCP.

Mohali, Punjab2–5 years1 questions5 answers

GCPDatabricksSQLPandasBigQuery

Streak: 11d
XP: 5868
Lessons done: 47
Studio passes: 35

Week Warrior25 Lessons25 Tickets5,000 XP

Share on X Share on LinkedIn

Questions

votes

answers

views

ADF copy activity is slow for many small Parquet files

Copy from ADLS Gen2 to Synapse: 40k small Parquet files (~2MB each) takes 6 hours. Same data coalesced to 200 files in Databricks runs in 25 minutes. Is ADF just the wrong tool here, or is this a config problem?

Azureazureadf