Apache Hudi: keys, indexes and table services

Keys, ordering and file groups

This is the main difference from Delta and Iceberg: Hudi tracks records by key, so it knows where a record lives before it writes. That makes it strong at high-volume upserts and incremental reads.

Copy-on-write and merge-on-read

Writing from Spark

hudi_options = {
    "hoodie.table.name": "customers",
    "hoodie.datasource.write.table.type": "MERGE_ON_READ",
    "hoodie.datasource.write.recordkey.field": "customer_id",
    "hoodie.datasource.write.precombine.field": "updated_at",   # the ordering field
    "hoodie.datasource.write.partitionpath.field": "country",
    "hoodie.datasource.write.operation": "upsert",
}
changes.write.format("hudi").options(**hudi_options).mode("append").save("s3://lake/customers")

Other write operations: insert (no key lookup, may create duplicates), bulk_insert (fast initial loads), delete, and insert_overwrite for partitions. The upsert path is where the index matters.

The index

To upsert, Hudi must find which file group already holds each incoming key. Scanning every file would be slow, so Hudi keeps an index:

The timeline and table services

Every action is an instant on the timeline in the .hoodie folder, with a time, a type (commit, deltacommit, compaction, clean, rollback, replacecommit for clustering) and a state (requested, inflight, completed). Readers only see completed instants, which gives atomic commits; a failed write is rolled back.

Services can run inline (in the writer, adding latency), asynchronously alongside a streaming writer, or as separate jobs. Skipping compaction on a MOR table is the most common cause of slow reads.

Common mistakes

• ✗No ordering field, or the wrong one — A late, older record can overwrite a newer one.
• ✗Using insert instead of upsert for CDC — insert skips the index lookup and can create duplicate keys.
• ✗Never compacting a MOR table — Every read merges an ever-growing pile of log files.
• ✗Choosing a bloom index for random UUID keys — Key ranges do not prune; most files are checked. Consider record-level or bucket indexes.