Spark 4.0 is a major release, and the change most likely to affect you is that ANSI SQL mode is on by default. Alongside it come new data types and SQL features, streaming improvements and a more mature Spark Connect.
ANSI mode on by default
Under ANSI rules, invalid operations fail loudly. Casting 'abc' to an integer, integer overflow, or division by zero now raise an error, where Spark 3 returned NULL or wrapped around. This is better for data quality, but it can break old jobs that relied on the lenient behaviour. You can switch back with spark.sql.ansi.enabled=false while you migrate.
New and improved features
- VARIANT type: a column type for semi-structured data such as JSON, stored in an efficient binary form, so you do not have to parse strings at every query or declare a rigid schema.
- SQL pipe syntax: write queries as a left-to-right chain with
|>, which reads like a DataFrame pipeline. - SQL scripting and session variables: control flow and variables inside SQL, so more logic can be written without leaving SQL.
- Python Data Source API: build custom batch or streaming data sources in pure Python.
transformWithState: a more flexible API for arbitrary stateful streaming logic, with better state handling than the older approach.- Spark Connect: wider API coverage, and a lighter Python client package.
- Platform changes: older Java versions (8 and 11) are no longer supported, so check your Java version, and the build uses Scala 2.13.
Migration risks
The first thing to run is your existing test suite against Spark 4, looking for new cast and overflow errors. Review jobs that parse strings into numbers or dates, and those that rely on NULL results from bad input. Libraries and connectors built for older Spark and Scala versions need updated releases.
What to say
Keep it short and honest: name ANSI default and VARIANT first, list two or three more, and say that you would roll out by running with ANSI on in test, fixing the failures, and using try_cast where you want tolerant parsing. Do not claim details of a feature you have not used.