11
Prefer a staging table plus validation gate before promoting to prod tables.
We have a Confluence space full of pipeline docs, most of which are 1-2 years stale and actively misleading at this point. What actually works to keep docs current, versus the usual "we'll keep it updated" that never happens?
Prefer a staging table plus validation gate before promoting to prod tables.
This matches our runbook for skewed keys.
Prefer a staging table plus validation gate before promoting to prod tables.
+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.
Docs colocated with code, a README next to the pipeline, docstrings, dbt model descriptions, survive far better than a separate wiki. The barrier to updating them during a PR is near zero, whereas remembering to also edit Confluence is a separate step people skip under deadline pressure.
Moving pipeline-specific docs into READMEs next to the code was the single biggest improvement for us. Confluence is now reserved for genuinely cross-cutting stuff only.
+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.
In our case the root cause was an implicit cast preventing pushdown.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.