Schema enforcement and evolution

Enforcement

A Delta or Iceberg write is checked against the table schema before commit. A plain ParquetParquet: A columnar file format: values of each column are stored together, so queries read only the columns they need. Learn more → folder accepts anything; a table format refuses:

This is a feature: a producer that suddenly sends amount as text fails loudly at write time instead of corrupting months of data.

Adding columns

# Let this write add new columns (Delta)
df.write.format("delta").mode("append").option("mergeSchema", "true").saveAsTable("events")

ALTER TABLE events ADD COLUMNS (device STRING, app_version STRING);

For MERGE, Delta supports automatic schema evolution with spark.databricks.delta.schema.autoMerge.enabled, and newer versions accept MERGE WITH SCHEMA EVOLUTION. Prefer explicit ALTER TABLE in production: schema changes become reviewed decisions instead of side effects of a write.

Which changes are safe

Why renames are hard, and how IDs fix them

Parquet files identify columns by name. If a table renames email to contact_email by metadata only, old files still say email, and a name-based reader would see the new column as null. Iceberg solved this from the start by giving every column a permanent field ID stored in the Parquet files; readers match by ID, so names can change freely. Delta added the same idea as column mapping:

spark.sql("""
    ALTER TABLE events SET TBLPROPERTIES (
      'delta.columnMapping.mode' = 'name',
      'delta.minReaderVersion' = '2',
      'delta.minWriterVersion' = '5')
""")
spark.sql("ALTER TABLE events RENAME COLUMN email TO contact_email")
spark.sql("ALTER TABLE events DROP COLUMN legacy_flag")

ALTER TABLE events SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.minReaderVersion' = '2',
  'delta.minWriterVersion' = '5');
ALTER TABLE events RENAME COLUMN email TO contact_email;
ALTER TABLE events DROP COLUMN legacy_flag;

Replacing the schema entirely

For incompatible changes, rewrite: df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable("events") in Delta, or CREATE OR REPLACE TABLE ... AS SELECT. This rewrites all data and keeps history, so time travel to the old schema still works within retention.

Common mistakes

• ✗Turning on mergeSchema everywhere — Any upstream typo becomes a permanent new column.
• ✗Renaming in Delta without column mapping — Not supported without it; enabling it changes the protocol.
• ✗Assuming all engines read evolved tables — Protocol upgrades can lock out older readers.