Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Delta schema enforcement and evolution

Snowflake, BigQuery & Databricks · Databricks

Delta schema enforcement and evolution

Mediumwarehouses-49
databricksdelta-lakeschema-enforcementschema-evolutioncolumn-mapping

Question

How does Delta Lake handle a write with a different schema?

Solution

Delta Lake checks every write against the table's schema, and rejects data that does not match. This is called schema enforcement. If you want the table to change, you opt in explicitly, which is called schema evolution.

Enforcement

If the table has columns (order_id BIGINT, amount DECIMAL) and you append a DataFrame that has an extra column coupon, or amount as a string, the write fails with an error. This protects the table from accidental corruption, such as a producer changing a field and silently polluting your data.

Evolution

To allow new columns on a write:

(df.write.format("delta")
   .mode("append")
   .option("mergeSchema", "true")
   .saveAsTable("sales.orders"))

With mergeSchema, new columns in the DataFrame are added to the table, and existing rows read NULL for them. For MERGE statements, you can set spark.databricks.delta.schema.autoMerge.enabled to do the same.

Breaking changes

To replace the table's schema completely during an overwrite, use overwriteSchema:

df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("sales.orders")

This is a heavy hammer, since it can discard the old structure, so use it deliberately.

Rename and drop

By default, renaming or dropping a column would require rewriting the data files. With column mapping enabled on the table, Delta separates the logical column name from the physical name in the files, so you can rename or drop columns as metadata-only operations.

Type widening

Delta can also widen some types in place, for example INT to BIGINT, without rewriting all the files, when the feature is enabled. Not all type changes are allowed, only the safe ones.

What to say

Enforcement is the default and is a good thing. Evolution is an opt-in for additive changes. For anything else, use column mapping or an explicit migration. Pair this with Auto Loader's rescued data column on the way in, so unexpected fields do not break ingestion.

PreviousNext