Data Engineering

Medallion Architecture: Building Bronze, Silver & Gold Data Layers

The Medallion Architecture is a data design pattern that organises data in a lakehouse into three progressively refined layers: Bronze (raw), Silver (cleaned & conformed), and Gold (business-ready). Coined and popularised by Databricks, it is now the de-facto standard for enterprise data lakes built on Delta Lake.

What is Medallion Architecture?

The core idea is simple: instead of landing data directly into a production-ready table, you pass it through a series of quality gates. Each layer serves a distinct purpose and audience:

• Bronze — raw, unmodified data as received from the source system.
• Silver — cleaned, validated, deduplicated, and schema-conformed data.
• Gold — aggregated, KPI-ready tables powering dashboards, ML features, and reporting.

Because each layer is a separate Delta Lake table, you can reprocess Silver from Bronze at any time — making the architecture both auditable and reprocessable without data loss.

Why It Matters

Without a layered approach, raw data gets mixed with curated data, quality issues propagate silently into production reports, and debugging a bad number means tracing it back through an undocumented transformation chain. Medallion Architecture solves this by:

• Separating ingestion from transformation — teams can work independently on each layer.
• Preserving raw data for audit, compliance, and reprocessing.
• Giving data consumers (analysts, data scientists) a stable, high-quality Gold layer to query.
• Enabling incremental processing — only new or changed records move through the pipeline.

Bronze Layer — Raw Ingestion

The Bronze layer ingests data exactly as it arrives from the source — no transformations, no schema enforcement. The goal is speed and completeness. You add metadata columns (ingestion timestamp, source file name, batch ID) but leave the payload untouched.

Here is a minimal PySpark example writing a CSV source to a Delta Bronze table in Azure Databricks:

from pyspark.sql import functions as F
from delta.tables import DeltaTable

# Read raw CSV from ADLS Gen2
raw_df = (
    spark.read
    .option("header", True)
    .option("inferSchema", True)
    .csv("abfss://raw@yourstorage.dfs.core.windows.net/orders/")
)

# Add ingestion metadata
bronze_df = raw_df.withColumn("_ingested_at", F.current_timestamp()) \
                  .withColumn("_source", F.lit("orders_csv"))

# Write to Delta Bronze table (append-only)
bronze_df.write \
    .format("delta") \
    .mode("append") \
    .option("mergeSchema", True) \
    .save("abfss://bronze@yourstorage.dfs.core.windows.net/orders/")

Notice mergeSchema=True: Bronze should accept schema evolution gracefully — if the upstream adds a column, you capture it rather than failing the pipeline.

Silver Layer — Cleanse & Conform

Silver applies the business rules: drop duplicates, enforce nullability, cast data types, standardise date formats, and filter out records that fail quality checks. A MERGE (upsert) pattern is common here so that corrections in Bronze propagate correctly to Silver.

from delta.tables import DeltaTable
from pyspark.sql import functions as F

# Read from Bronze
bronze = spark.read.format("delta") \
    .load("abfss://bronze@yourstorage.dfs.core.windows.net/orders/")

# Clean and conform
silver_df = (
    bronze
    .dropDuplicates(["order_id"])
    .filter(F.col("order_id").isNotNull())
    .withColumn("order_date", F.to_date("order_date", "yyyy-MM-dd"))
    .withColumn("amount", F.col("amount").cast("decimal(18,2)"))
    .select("order_id", "customer_id", "order_date", "amount", "status")
)

# MERGE into Silver Delta table
silver_table = DeltaTable.forPath(
    spark, "abfss://silver@yourstorage.dfs.core.windows.net/orders/"
)

silver_table.alias("s").merge(
    silver_df.alias("u"),
    "s.order_id = u.order_id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

The MERGE ensures idempotency — running the pipeline twice does not create duplicates.

Gold Layer — Business-Ready Aggregates

Gold tables are purpose-built for consumption. They may be wide denormalised tables optimised for Tableau, or narrow aggregation tables that back a specific KPI card. You typically use Spark SQL or DataFrame aggregations here.

# Daily revenue summary — Gold
gold_df = spark.sql("""
    SELECT
        order_date,
        status,
        COUNT(order_id)           AS total_orders,
        SUM(amount)               AS total_revenue,
        AVG(amount)               AS avg_order_value
    FROM delta.`abfss://silver@yourstorage.dfs.core.windows.net/orders/`
    WHERE status = 'COMPLETED'
    GROUP BY order_date, status
    ORDER BY order_date DESC
""")

gold_df.write \
    .format("delta") \
    .mode("overwrite") \
    .option("overwriteSchema", True) \
    .save("abfss://gold@yourstorage.dfs.core.windows.net/daily_revenue/")

Because Gold is derived and deterministic, a full overwrite on each run is acceptable — unlike Bronze and Silver where you need append or merge to preserve history.

Unity Catalog & Governance

In a production Azure Databricks setup, register all three layers in Unity Catalog. This gives you:

• Column-level access control — mask PII in Bronze but expose it only to authorised Silver consumers.
• Data lineage — Unity Catalog tracks which Gold table descended from which Silver and Bronze sources.
• Audit logs — every query on every table is logged, satisfying GDPR and SOC 2 requirements.

Pair Unity Catalog with dbt for the Silver-to-Gold transformations: dbt handles dependency graphs, documentation, and data quality tests natively.

When to Use Medallion Architecture

Medallion is the right choice when:

• You have multiple upstream source systems writing to a central lake.
• Data quality is inconsistent and you need an auditable cleansing record.
• Multiple teams (analytics, data science, reporting) share the same data with different quality requirements.
• Regulatory or compliance requirements mandate raw data retention.

It is overkill for a single-source, low-volume use case where a simple ETL into a warehouse table suffices. Start with Medallion when you expect more than 3–4 source systems or complex data quality rules.