22
How do you handle backfill without duplicating rows?
I have read the docs but real-world tradeoffs are unclear. What would you optimize first?
Context: Snowflake warehouse auto-suspend vs always-on dev cost
Happy to share schema snippets or metrics if useful.
How do you handle backfill without duplicating rows?
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
In our case the root cause was an implicit cast preventing pushdown.
Agree on the staging table swap. Atomic promote prevented partial reads.
+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.
Document the grain decision, most BI bugs turn out to be grain bugs.
Idempotent writes with merge keys saved us during backfills.
Prefer a staging table plus validation gate before promoting to prod tables.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.