Medallion Architecture in Practice

You’ve got raw log files landing every minute, API extracts piling up in cloud storage, and business users asking for dashboards that always show the latest numbers. If you build one giant table and call it done, you’ll drown in schema changes, corrupted records, and reprocessing nightmares. That’s exactly the pain the medallion architecture solves — a simple Bronze → Silver → Gold pattern that turns chaos into a reliable, governable data pipeline. In this lesson, you’ll learn to build a medallion architecture in practice on Databricks, step by step, with code you can run today.

The problem this lesson solves

Unstructured, ever-changing data is the default state of most ingestion pipelines. Users expect clean, curated datasets for reporting, but the raw data is full of surprises: missing fields, type mismatches, duplicate IDs, and files that arrive late or out of order. Without a formal structure, your team ends up with spaghetti pipelines where every consumer writes their own parsing logic — and any schema change breaks everything downstream.

The medallion architecture directly addresses these pain points. It gives you a staged approach to data transformation, where each layer has a clear purpose and a clear contract. You stop mixing raw ingestion logic with business rules, and you stop reprocessing the entire history every time a transformation changes. Instead, you build a pipeline that is incremental, auditable, and resilient. As a Data Engineer on Databricks, this is the pattern you’ll rely on for almost every production workload.

Core concept / mental model

Think of the medallion architecture like a food preparation kitchen. Bronze is the delivery dock — you receive all ingredients exactly as they arrive, in their original packaging, no matter how messy. Silver is the prep station — you wash, peel, chop, and standardize the ingredients, putting each one in a labeled container. Gold is the final plating — you combine prepared ingredients into specific dishes that are ready for the customer (your dashboard, ML model, or analyst).

The three layers map to three distinct table sets on Databricks, typically implemented as Delta Lake tables:

• Bronze: Raw data as ingested. Append-only, immutable. Includes all the messiness — duplicate rows, malformed values, and unexpected columns.
• Silver: Cleaned and validated data. Deduplicated, type-corrected, conformed to a schema. This is the "source of truth" for analytics.
• Gold: Aggregated, business-level tables (e.g., daily sales, customer KPIs). Optimized for consumption by BI tools and dashboards.

Pro tip: Don't overthink the naming. Bronze, Silver, Gold is a mental model, not a strict standard. The key is that each layer has a single responsibility and a stable contract with the next.

This layered approach makes your pipeline incremental — you only process new data in each layer — and reusable — the same Bronze data feeds multiple Silver transformations, and the same Silver data feeds many Gold aggregates. It also gives you auditability, because you can always trace a Gold number back to the exact Bronze record.

How it works step by step

Here’s the high-level flow of building a medallion architecture on Databricks:

1. Ingest raw data into Bronze — Use Auto Loader or Spark structured streaming to read from cloud storage (S3, ADLS, GCS). Write as Delta tables, append-only, with _rescued_data to capture schema drift.
2. Validate and clean into Silver — Read Bronze, apply transformations: deduplication, type casting, filtering, and joining with reference data. Write to Silver as Delta tables, typically using MERGE for upserts.
3. Aggregate into Gold — Read Silver, compute business metrics (sums, counts, window functions), and write compact, highly-optimized tables for reporting.
4. Automate — Schedule the pipeline as a Databricks Job, or use Delta Live Tables (DLT) to declare the pipeline declaratively.

The key to making this work is incremental processing. Bronze is append-only, so Silver can process only new records using structured streaming or AUTO_INCREMENT IDs. Gold is built on top of Silver, so it only recomputes the affected partitions. This keeps your pipeline fast and your costs low.

Hands-on walkthrough

Let’s build a real pipeline. Assume you have JSON clickstream events streaming into s3://my-bucket/events/. We’ll use Python with PySpark on Databricks, but the same logic applies in Scala or SQL.

Step 1: Ingest to Bronze

We use Auto Loader to incrementally load JSON files into a Bronze Delta table.

from pyspark.sql.functions import col, current_timestamp, input_file_name

# Define paths
bronze_path = "/mnt/analytics/bronze/clickstream"
checkpoint_path = "/mnt/analytics/checkpoints/clickstream_bronze"

# Read streaming JSON from cloud storage
stream_df = (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/mnt/analytics/schemas/clickstream")
    .option("cloudFiles.inferColumnTypes", "true")
    .load("/mnt/analytics/raw/clickstream")
)

# Add metadata columns and write append-only
query = (
    stream_df
    .withColumn("ingest_time", current_timestamp())
    .withColumn("source_file", input_file_name())
    .writeStream
    .format("delta")
    .option("checkpointLocation", checkpoint_path)
    .outputMode("append")
    .table("clickstream_bronze")
)

query.awaitTermination()

Expected output: A Delta table clickstream_bronze that grows as new files land. Every record includes an ingest_time and source_file for lineage.

Step 2: Clean into Silver

Now read from Bronze and apply cleaning logic. We deduplicate, cast types, and drop invalid events.

# Read Bronze table
bronze_df = spark.table("clickstream_bronze")

# Example cleaning transformations
silver_df = (
    bronze_df
    .dropDuplicates(["event_id"])
    .filter(col("user_id").isNotNull())
    .withColumn("event_timestamp", col("event_timestamp").cast("timestamp"))
    .withColumn("session_id", col("session_id").cast("string"))
)

# Write to Silver with merge for upserts
from delta.tables import DeltaTable

target = DeltaTable.forName(spark, "clickstream_silver")

merge_query = (
    target.alias("t")
    .merge(silver_df.alias("s"), "t.event_id = s.event_id")
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
)
merge_query.execute()

Expected output: clickstream_silver contains only unique, valid events with correct types. You can query it as SELECT * FROM clickstream_silver and see clean data.

Step 3: Aggregate into Gold

Finally, compute user session aggregates for reporting.

from pyspark.sql import functions as F

silver_df = spark.table("clickstream_silver")

gold_df = (
    silver_df
    .groupBy("user_id", F.window("event_timestamp", "1 hour"))
    .agg(
        F.count("*").alias("events_count"),
        F.sum("page_views").alias("page_views")
    )
)

# Write as a Delta table, optimized for BI
gold_df.write.format("delta").mode("overwrite").saveAsTable("clickstream_hourly_aggregates")

Expected output: A Gold table with hourly audience metrics, ready to connect to Power BI or Tableau.

Complete pipeline automation with Delta Live Tables (optional)

If you want to declare the entire pipeline as code, DLT is the modern approach:

import dlt
from pyspark.sql.functions import col, current_timestamp, input_file_name

@dlt.table
@dlt.expect_all_valid({"valid_user": "user_id IS NOT NULL"})
def clickstream_bronze():
    return (
        spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/mnt/analytics/raw/clickstream")
        .withColumn("ingest_time", current_timestamp())
    )

@dlt.table
def clickstream_silver():
    return (
        dlt.read("clickstream_bronze")
        .dropDuplicates(["event_id"])
        .select("event_id", "user_id", "event_timestamp", "page_views")
    )

@dlt.table
def clickstream_hourly_aggregates():
    return (
        dlt.read("clickstream_silver")
        .groupBy("user_id", ...)
        .agg(...)
    )

Run this as a DLT pipeline in the UI — Databricks handles dependencies, scheduling, and data quality checks for you.

Compare options / when to choose what

Pro tip: If you’re starting fresh, use DLT. It enforces the medallion pattern by design and saves you hours of orchestration code. But if you need granular control over every transformation, manual Spark jobs give you that flexibility.

Troubleshooting & edge cases

• Schema drift — New fields arrive in Bronze that aren’t in Silver’s schema. Fix: use _rescued_data column to capture unexpected fields, then explicitly cast or include them in Silver logic. Don’t try to make Silver dynamic; keep it tightly controlled.
• Deduplication fails — You drop duplicates on event_id, but you still see duplicates. Fix: check if the event_id is null — drop nulls first. Or your dropDuplicates is on wrong columns (e.g., date + user_id vs unique event ID). Inspect with groupBy().count().filter('count > 1').
• Merge doesn’t update — You expect merge to update existing rows, but old values persist. Fix: ensure your merge condition matches a unique key. If the key isn’t unique, whenMatchedUpdateAll can overwrite with any duplicate, leading to nondeterminism.
• Auto Loader reads files twice — Checkpoint path is critical. If you change the checkpoint path, Auto Loader will reprocess all files. Always keep the checkpoint path stable.
• Performance issues in Gold — If aggregate queries are slow, ensure you’ve OPTIMIZEd and ZORDER BY the Gold table on common filter columns like date. Also consider using MERGE for incremental upserts rather than full overwrite.

What you learned & what's next

You now understand the problem the medallion architecture solves, the mental model of Bronze, Silver, and Gold, and how to implement it step by step with Auto Loader, Delta Lake, and optional DLT. You’ve completed a practical exercise that ingests raw JSON, cleans it, and aggregates it for reporting. You can explain the core idea behind the medallion pattern and apply it to your own pipelines.

Next in the Databricks learning path, you’ll dive into incremental data processing — mastering how to design streaming and batch pipelines that only touch new data, which is the engine that makes the medallion architecture fast and cost-efficient.