Skip to content
Great engineers knowNEW 3 min read · Delta

Delta Lake 4.0

VARIANT, type widening, collations and the other changes that shipped alongside Spark 4.0.

On this page

Show code in

Every code block on the page follows this.

You will learn

  • The headline features of Delta Lake 4.0
  • How VARIANT, type widening and collations work in Delta
  • What Delta Connect and catalog-managed commits are for
  • How to upgrade safely, including protocol features

Read first

Comfortable with these? Read on.

TL;DR Delta Lake 4.0, released for Apache Spark 4.0, adds VARIANT columns, type widening, collations, Delta Connect (Delta over Spark Connect) and moves commit coordination toward catalogs. Most new features are opt-in table features that raise the table protocol.

Headline features

FeatureWhat it does
VARIANTStore semi-structured data in a variant column using Spark 4.0's binary encoding, instead of JSON strings.
Type wideningChange a column to a wider type (for example int to long, float to double, decimal precision increases) without rewriting data.
CollationsColumns with collation-aware comparison and sorting, such as case-insensitive or locale-specific string ordering.
Delta ConnectUse Delta APIs (DeltaTable, merge, history) from Spark Connect clients.
Coordinated / catalog-managed commitsLet a catalog or coordinator, rather than the storage system alone, order commits: the basis for multi-table transactions and catalog-level governance.
Identity columnsAuto-generated unique values for new rows.

Feature names, maturity (preview vs generally available) and details vary across 4.x releases; read the release notes of the exact version you deploy.

VARIANT in Delta

Spark SQL 
CREATE TABLE events (id BIGINT, ts TIMESTAMP, payload VARIANT) USING delta;
INSERT INTO events SELECT id, ts, parse_json(raw_json) FROM landing;

SELECT variant_get(payload, '$.device.os', 'string') AS os, COUNT(*)
FROM events GROUP BY 1;

Adding a variant column enables the variantType table feature, so readers must support it.

Type widening

Spark SQL 
ALTER TABLE orders SET TBLPROPERTIES ('delta.enableTypeWidening' = 'true');
ALTER TABLE orders ALTER COLUMN quantity TYPE BIGINT;

Old files keep the narrow type on disk; readers widen values as they read. With type widening enabled, MERGE and appends with schema evolution can also widen types automatically. This solves a classic pain point: an INT column that overflows, which previously required rewriting the whole table.

Why commit coordination matters

Until now a Delta commit was defined purely by creating the next file in _delta_log, which needs storage with put-if-absent, makes multi-table transactions impossible, and lets anyone with storage access write commits outside the catalog's control. Moving commit ordering to a coordinator or catalog lets a catalog enforce permissions on writes, coordinate multi-table transactions and work on any storage, similar in spirit to how Iceberg commits through its catalog.

Upgrading safely

  1. 1
    Upgrade the Delta library together with Spark 4.0; Delta 4.0 targets Spark 4.0.
  2. 2
    Existing tables keep working; nothing changes until you enable new features.
  3. 3
    Enabling a feature (variant, type widening, collations) adds a table feature and raises the protocol. Check that every reader (other Spark versions, Trino, other connectors) supports it first.
  4. 4
    Remember Spark 4.0 also turns on ANSI mode by default, which can surface new errors in existing jobs.

Common mistakes

Enabling new table features before checking readers

Older clients can no longer read the table.

Assuming every 4.0 feature is GA

Some ship as previews; check release notes.

Upgrading Delta without Spark

Delta 4.0 is built for Spark 4.0.

Key takeaways

  • Delta 4.0 targets Spark 4.0 and adds VARIANT, type widening and collations.
  • Delta Connect brings Delta APIs to Spark Connect clients.
  • Commit coordination is moving toward catalogs.
  • New features are opt-in table features that can lock out older readers.

Check yourself

3 questions

1. What does type widening allow?

Show the answer

Changing int to long without rewriting data. Old files are widened on read.

2. What happens to a table's protocol when you add a VARIANT column?

Show the answer

A table feature is added, so readers must support it. Variant support is a table feature.

3. What is Delta Connect?

Show the answer

Delta APIs usable from Spark Connect clients. It extends Spark Connect with Delta operations.

Go deeper

Spark internals
The VARIANT typeLakehouse
Schema enforcement and evolutionSpark internals
Spark Connect

Primary sources: Delta Lake releases · Delta Lake documentation

← Previous
Iceberg format v3