Data lake storage: formats and layout

What a data lake is

A data lake stores data as files in cheap object storage (Amazon S3, Azure Data Lake Storage, Google Cloud Storage) and lets many engines read it: Spark, Trino, Flink, DuckDB, warehouses. Storage is separated from compute, so you pay for compute only while queries run. The price is that the lake has no database to enforce anything: formats, layout and conventions do that job, and table formats like Delta and Iceberg add transactions on top.

The formats compared

Why columnar wins for analytics

• Column pruningcolumn pruning: Reading only the columns a query uses, which columnar files like Parquet make possible. Learn more →: SELECT sum(amount) reads one column out of fifty, not all of them.
• Better compression: similar values sit together, so dictionary and run-length encoding shrink them a lot (a city column with 100 distinct values compresses to almost nothing).
• Skipping: each row grouprow group: A block of rows inside a Parquet file. Its min and max statistics let readers skip blocks that cannot match a filter. Learn more → stores min/max statistics per column, so WHERE amount > 1000 skips row groups whose max is lower.
• The cost: writing is heavier, and reading one whole record touches every column chunk. That is why landing and streaming often use row formats.

Compression codecs

Inside Parquet, compression applies per column chunk, so every codec is splittable there. Outside Parquet, a single 10 GB .csv.gz file is read by one task, which is a common reason a job has one task running for an hour.

Zones: raw to curated

1. Raw / landing

   Exactly as received: CSV, JSON, Avro. Append-only, never edited, kept for replay and audit.

2. →

   Cleaned / bronze-silver

   Parsed, typed, de-duplicated, in Parquet or a table format.

3. →

   Curated / gold

   Modelled for consumers: facts, dimensions, aggregates.

This is the same idea as the medallion architecture. Keeping the raw zone means any bug downstream can be fixed by reprocessing from the original files.

Folder layout

s3://acme-lake/
  raw/orders/ingest_date=2025-03-01/part-0000.json.gz
  silver/orders/order_date=2025-03-01/part-00000.zstd.parquet
  gold/daily_revenue/...

• Hive-style folders (column=value) let engines prune partitions from the path.
• Partition by a low-cardinalitycardinality: How many distinct values a column has. User ids are high cardinality; country codes are low. Learn more → column that queries filter on, usually a date. Partitions should hold at least about 1 GB.
• Aim for files of 128 MB to 1 GB.
• Separate buckets or prefixes per zone make access control simple: few people can write to raw, many can read gold.
• Never rely on listing folders to know what a table contains: concurrent writers make that unsafe. That is the problem table formats solve with a transaction logtransaction log: The ordered list of commits that defines which files make up a Delta table at each version. Learn more →.

Common mistakes