13
Prefer a staging table plus validation gate before promoting to prod tables.
Copy from ADLS Gen2 to Synapse: 40k small Parquet files (~2MB each) takes 6 hours. Same data coalesced to 200 files in Databricks runs in 25 minutes.
Is ADF just the wrong tool here, or is this a config problem?
Prefer a staging table plus validation gate before promoting to prod tables.
Plain English: the system prefers to guess a good-enough plan than the perfect plan, because figuring out the perfect plan would take longer than just running the good-enough one.
ADF's per-file overhead really hurts at 40k files. Compact upstream, or use Databricks/Spark for copying many small files.
Careful with NULL in join keys, they'll drop rows in an inner join.
Sign in to reply.