Catalogs: Unity, Polaris, Iceberg REST
What a catalog does for a lakehouse, and the open catalogs that arrived in 2024 and 2025.
On this page
You will learn
- What a catalog does in a lakehouse
- The main catalogs: Hive Metastore, Glue, Unity Catalog, Polaris, Iceberg REST
- What the Iceberg REST protocol standardises
- Why open catalogs became the big story of 2024 and 2025
Read first
- Open table formats · 3 min read
- Iceberg metadata tree · 4 min read
Comfortable with these? Read on.
What a catalog does
- Names:
sales.ordersresolves to a storage location and table format. - Commits: for Iceberg, the catalog performs the atomic pointer swap that makes a commit; it is part of the transaction path.
- Governance: permissions on catalogs, schemas, tables, columns and rows; audit logs; lineage.
- Credential vending: instead of giving every engine broad storage keys, the catalog hands out short-lived, table-scoped credentials.
- Discovery: search, comments, tags and ownership.
The main catalogs
| Catalog | What it is |
|---|---|
| Hive Metastore | The original Hadoop catalog, a service backed by a database. Universally supported, but no fine-grained governance and not designed for table formats. |
| AWS Glue Data Catalog | Managed, Hive-compatible catalog on AWS, with Iceberg support and Lake Formation for permissions. |
| Unity Catalog | Databricks' governance catalog; open-sourced in June 2024 (Linux Foundation AI & Data). Supports Delta, Iceberg through a REST API, and also volumes, functions and models. |
| Apache Polaris (incubating) | An open-source implementation of the Iceberg REST catalog, launched by Snowflake in 2024 and donated to Apache. |
| Project Nessie | Git-like catalog for Iceberg with branches, tags and multi-table commits. |
| Iceberg REST catalog | Not a product but a protocol: an OpenAPI spec any catalog can implement. |
The Iceberg REST protocol
Before REST, every engine needed a client library for every catalog backend (Hive, Glue, JDBC, Nessie...). The REST spec defines HTTP endpoints for listing namespaces and tables, loading table metadata and committing changes, so an engine that speaks REST works with any compliant catalog. The server can also vend credentials and perform server-side commit validation.
spark = (SparkSession.builder
.config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.lake.type", "rest")
.config("spark.sql.catalog.lake.uri", "https://catalog.example.com/api/catalog")
.config("spark.sql.catalog.lake.warehouse", "analytics")
.getOrCreate())
spark.table("lake.sales.orders").show()
Why it matters
Open table formats made data portable, but the catalog still decided which engines could find and write tables, and enforced permissions in one vendor's system. With open catalogs implementing a shared protocol, the same governed tables can be read and written by Spark, Trino, Flink, Snowflake, Databricks and others. The competitive focus has moved from formats to catalogs and governance.
Common mistakes
Treating the catalog as just a name lookup
Sharing long-lived storage keys with every engine
Registering one table in two writable catalogs
Key takeaways
- A catalog resolves names, performs commits (Iceberg), governs access and vends credentials.
- Iceberg REST is a protocol any catalog can implement.
- Unity Catalog (open-sourced 2024) and Apache Polaris implement it.
- One writable catalog should own each table.
Check yourself
3 questions1. What role does the catalog play in an Iceberg commit?
Show the answer
It atomically swaps the pointer to the new metadata file. The pointer swap in the catalog is the atomic step.
2. What is the Iceberg REST catalog?
Show the answer
An open protocol (OpenAPI spec) for catalog operations. It standardises how engines talk to catalogs.
3. What is credential vending?
Show the answer
The catalog issuing short-lived, scoped storage credentials to engines. Engines get temporary access only to the tables they are allowed to use.
Go deeper
Primary sources: Iceberg REST catalog spec · Unity Catalog (open source) · Apache Polaris