3
Accepted answer
Streaming checkpoints store the query plan, which is schema-sensitive. Adding a column upstream while the checkpoint metadata still references the old schema is a common source of resume failures, not specific to Event Hubs at all.
Structured Streaming job reading from Event Hubs into a Delta table failed to resume from checkpoint after we added a new optional field to the event schema. Had to delete the checkpoint and reprocess from the retention window.
Accepted answer
Streaming checkpoints store the query plan, which is schema-sensitive. Adding a column upstream while the checkpoint metadata still references the old schema is a common source of resume failures, not specific to Event Hubs at all.
Document the grain decision, most BI bugs turn out to be grain bugs.
Prefer a staging table plus validation gate before promoting to prod tables.
Simple way to think about it: caching is a bet that you'll ask the same question again soon. If you don't, you're just paying rent on memory for nothing.
Prefer a staging table plus validation gate before promoting to prod tables.
Idempotent writes with merge keys saved us during backfills.
Worth noting: a truly additive, nullable field usually shouldn't require a checkpoint reset. If it did here, double check whether the writeStream's outputMode or aggregation logic also changed at the same time, not just the source schema.
In simple terms: your job is asking for way more data than it actually needs, and that extra data has to travel across the network or between machines, which is slow. Ask for less, or ask smarter.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.