Producer added a new required field to an Avro schema. Registry accepted it under FORWARD compatibility, but an older consumer using an outdated schema failed to deserialize the new messages entirely.
Accepted answer
FORWARD compatibility only guarantees new producer schemas can be read by consumers using the previous schema. It doesn't cover consumers running arbitrarily old schemas several versions behind. A required, no-default field is the actual break here.
+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.
Consider DuckDB or Polars for this size before spinning up a cluster.
What batch size or interval worked for you at similar scale?
Made the field optional with a default. Also switched to BACKWARD_TRANSITIVE compatibility to catch this class of gap earlier next time.
New fields should almost always be optional with a default value regardless of compatibility mode. That's what actually keeps old consumers working, not the mode setting on its own.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.
