The medallion architecture
Bronze, silver and gold layers: what belongs in each, and where the pattern goes wrong.
On this page
Show code in
Every code block on the page follows this.
You will learn
- What the bronze, silver and gold layers hold
- Which transformations belong in each layer
- How data flows and is reprocessed between layers
- Where the pattern goes wrong
The three layers
| Layer | Holds | Typical operations | Consumers |
|---|---|---|---|
| Bronze | Raw data, as close to the source as possible, plus ingestion metadata | Append-only loads, minimal parsing, add load timestamp and source file | Data engineers, reprocessing |
| Silver | Validated, deduplicated, typed and joined data at the entity level | Type casting, deduplication, MERGE of CDC, conforming keys, quality checks | Engineers, analysts, data scientists |
| Gold | Business-ready tables: facts and dimensions, aggregates, features | Aggregations, business logic, star schemas | BI dashboards, reports, ML, applications |
Walk through one pipeline
Bronze · land raw
bronze.orders_raw exactly as received, with _ingested_at and _source columns. Malformed records are kept, not dropped: bronze is your replayable history.The command
raw = spark.readStream.format("kafka").option("subscribe", "orders").load() (raw.selectExpr("CAST(value AS STRING) AS json", "timestamp AS _ingested_at") .writeStream.option("checkpointLocation", ckpt).toTable("bronze.orders_raw"))
Silver · clean and merge
order_id keeping the latest event, and MERGE into silver.orders so it holds one current row per order.The command
MERGE INTO silver.orders t USING latest_order_events s ON t.order_id = s.order_id WHEN MATCHED AND s.event_ts > t.event_ts THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *
Gold · serve
gold.daily_revenue by date and region, and gold.fct_orders joined to customer and product dimensions. These are the tables dashboards query.The command
CREATE OR REPLACE TABLE gold.daily_revenue AS SELECT order_date, region, SUM(amount) AS revenue, COUNT(*) AS orders FROM silver.orders JOIN silver.customers USING (customer_id) GROUP BY order_date, region
Why layers help
- Reprocessing: when silver logic has a bug, rebuild silver from bronze instead of re-extracting from source systems.
- Clear contracts: consumers know gold is curated and stable; bronze carries no guarantees.
- Separation of concerns: ingestion, cleaning and business logic change at different speeds and are owned by different people.
- Debugging: you can compare each layer to find where a number went wrong.
Where it goes wrong
Layers as a rule, not a tool
Business logic in bronze
Gold tables that copy silver
No retention on bronze
Common mistakes
Dropping malformed records at ingestion
Letting dashboards read silver directly
Ignoring idempotency
Key takeaways
- Bronze keeps raw data, silver cleans and conforms it, gold serves the business.
- Bronze is the replayable history; do not transform it destructively.
- Silver is where deduplication and MERGE of changes happen.
- Use layers where they add value, and make each step idempotent.
Check yourself
3 questions1. Where should malformed source records go?
Show the answer
Kept in bronze or a quarantine table. Bronze preserves raw data so problems can be investigated and replayed.
2. Which layer typically holds deduplicated entity tables updated by MERGE?
Show the answer
Silver. Silver is the cleaned, conformed, entity-level layer.
3. Main benefit of keeping bronze?
Show the answer
Rebuilding downstream tables without re-extracting from sources. Raw history lets you rerun silver and gold logic after fixes.
Practice it
Interview problems that use this: write the PySpark, run it, and get graded on hidden tests.
Go deeper
Primary sources: Delta Lake: best practices