Photon is Databricks' own query engine, written in C++ instead of running on the JVM. It executes Spark SQL and DataFrame operations faster, especially scans, joins and aggregations on Delta and Parquet data, with no changes to your code.
How it works
Spark's standard engine runs on the Java Virtual Machine. Photon replaces parts of the physical plan with native, vectorized code that processes data in batches of values, using the CPU's modern instructions and good cache behaviour. When Photon supports an operator, it runs there. When it does not, the plan falls back to the regular Spark engine for that step, and you can see mixed plans in the Spark UI, where Photon operators have a Photon label.
Turning it on
It is a setting when you create a cluster (choose a Photon-enabled runtime), and it is built into SQL warehouses and serverless compute. You do not change any queries.
Where it helps most
- Large scans with filters and aggregations on columnar data.
- Joins and sorts on big tables.
- Writes and updates to Delta tables, such as MERGE.
- SQL warehouse dashboards with many queries.
Where it helps little
- Python UDFs. They run in a Python process regardless, so Photon cannot speed them up. Prefer built-in functions.
- RDD code and operations that Photon does not support.
- Very small jobs that are limited by startup time or by waiting on external systems.
Cost
Photon-enabled compute has a higher DBU rate. The question is whether the speedup is bigger than the price increase. For heavy SQL transformations the answer is often yes, since jobs finish in a fraction of the time. For a job dominated by a Python UDF, you pay more for no gain.
How to decide
Run the same job with and without Photon, and compare the total cost (runtime times rate), not only the runtime. That measurement is what an interviewer wants to hear.