Predictive optimization lets Databricks run table maintenance for you. It automatically decides when to run OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, based on how the tables are used, so you do not schedule maintenance jobs yourself.
What problem it solves
Without it, someone has to write and schedule jobs that compact files, clean up old data and refresh statistics, tune how often they run for every table, and pay for the cluster that does it. These jobs are easy to forget, and they often run on tables that do not need them, or too late on tables that do.
How it works
Databricks observes things such as how many small files a table has, how it is queried, and how much data could be reclaimed. It then runs the right operation on its own compute when it is worthwhile, and skips tables where it would not help. It runs on serverless compute, and it is billed as its own line item.
ALTER SCHEMA prod.sales ENABLE PREDICTIVE OPTIMIZATION;
You can enable it at the account, catalog or schema level. For newer accounts it may already be on by default, so check your settings.
Where it applies
It works on Unity Catalog managed tables. External tables are not covered, which is one more reason to prefer managed tables when you can. It also works together with liquid clustering. With automatic liquid clustering, Databricks can even choose and change the clustering keys based on query patterns, rather than you picking them.
Trade-offs
- You give up some control over exactly when maintenance happens.
- It costs money, billed as serverless usage, so watch the system tables for its charges. In most cases the cost is lower than the manual jobs plus the slow queries they prevent.
- It does not fix bad data design. A table partitioned in a poor way, or fed by a stream that writes thousands of tiny files every minute, still deserves attention at the source.
In an interview, say it replaces routine maintenance jobs, and that you would still review the file counts and costs now and then.