The VARIANT type
Store and query semi-structured JSON efficiently without declaring a schema up front.
On this page
Show code in
Every code block on the page follows this.
You will learn
- What the VARIANT type is and why Spark added it
- How to parse, query and cast VARIANT data
- How it compares with JSON strings and structs
- Where it is supported in the lakehouse
parse_json and read fields with variant_get(v, '$.path', 'type'), without declaring a schema, and much faster than re-parsing JSON strings.The problem
Event payloads, API responses and IoT messages have shapes that differ by event type and change over time. Until now you had two poor options:
Store as a JSON string
from_json or get_json_object, which is slow, and no statistics can skip data.Declare a struct schema
VARIANT keeps the flexibility of a string with most of the speed of a struct.
Using it
from pyspark.sql import functions as F ev = raw.select(F.parse_json("payload").alias("v")) ev.select( F.variant_get("v", "$.user.id", "long").alias("user_id"), F.variant_get("v", "$.items[0].sku", "string").alias("first_sku"), F.try_variant_get("v", "$.price", "decimal(10,2)").alias("price"), )
SELECT variant_get(v, '$.user.id', 'long') AS user_id, variant_get(v, '$.items[0].sku', 'string') AS first_sku, try_variant_get(v, '$.price', 'decimal(10,2)') AS price FROM (SELECT parse_json(payload) AS v FROM raw)
| Function | Does |
|---|---|
parse_json(str) | JSON text to VARIANT; errors on invalid JSON |
try_parse_json(str) | Same, null on invalid JSON |
variant_get(v, path, type) | Extract and cast a field; errors if the cast fails |
try_variant_get(...) | Same, null if the cast fails |
schema_of_variant(v) | Describe the shape of a value |
to_json(v) | Back to JSON text |
How it is stored
A VARIANT value is binary: a dictionary of field names plus an encoded value with type tags, so reading a nested field means following offsets instead of scanning and parsing characters. The open specification also defines shredding: frequently used fields can be stored as regular typed Parquet columns alongside the binary value, so readers get column pruning and min/max statistics for them.
Where it works
- Apache Spark 4.0 (SQL and DataFrame APIs).
- Delta Lake 4.0 tables, through the variant table feature.
- Apache Iceberg format v3, which adds a variant type to the table spec.
- The binary encoding is specified in the Apache Parquet project, so other engines can implement the same format.
When to use which
| Data | Best type |
|---|---|
| Stable, well-known fields queried often | Struct columns (or promote fields to top-level columns) |
| Varying or evolving shapes, sparse fields | VARIANT |
| Text you only store and return, never query | String is fine |
Common mistakes
Using variant_get without a type
Expecting VARIANT on older engines
Putting everything in one VARIANT
Key takeaways
- VARIANT stores semi-structured data in a binary format, new in Spark 4.0.
- parse_json to load; variant_get(v, path, type) to read and cast.
- Faster than JSON strings, more flexible than structs.
- Supported in Delta Lake 4.0 and Iceberg v3; check reader compatibility.
Check yourself
3 questions1. Which function reads $.user.id from a VARIANT as a long?
Show the answer
variant_get(v, '$.user.id', 'long'). variant_get takes the path and a target type.
2. Why is VARIANT faster than a JSON string for queries?
Show the answer
Fields are found by following offsets in a binary encoding instead of parsing text. The binary layout avoids re-parsing text on every query.
3. What does try_variant_get return when the value cannot be cast?
Show the answer
null. The try_ variant returns null instead of failing.
Go deeper
Primary sources: Spark SQL built-in functions · Parquet variant encoding