Run Delta Live Tables Pipeline

You've written your Delta Live Tables (DLT) code, validated your notebooks, and tested your transformations in development. Now comes the moment of truth: running a Delta Live Tables pipeline in Databricks. Without this step, your brilliant data engineering work is just dormant code. The painful reality is that a poorly executed pipeline run can lead to silent data corruption, hours of debugging, and frustrated stakeholders waiting for fresh data. In this lesson, you'll learn how to run a delta live tables pipeline confidently, from configuration to monitoring, and we'll tackle the common pitfalls so you can deploy with certainty.

The problem this lesson solves

So you've built a multi-stage DLT pipeline with Bronze, Silver, and Gold tables. You click Start on the pipeline UI, and… nothing. Or maybe it runs but fails halfway through with a cryptic error. Or worse, it succeeds but the data is wrong. The gap between writing DLT code and having a reliable, production-ready data flow is one of the biggest hurdles in data engineering.

Running a pipeline isn't just about clicking a button. You need to understand:

• How DLT pipelines are configured for different environments (development vs. production).
• How to trigger a run and what happens during each execution.
• How to monitor the run and interpret logs and metrics.
• How to recover from failures without tearing your hair out.

This lesson solves that gap. By the end, you'll be able to launch a pipeline run with confidence, know what to expect, and know how to react when things go sideways.

Core concept / mental model

Think of a DLT pipeline as an assembly line for your data. Each DLT table is a workstation, and the pipeline is the manager that ensures raw materials (source data) flow through each station in the correct order, with quality checks at every step. The pipeline engine (built on Apache Spark) orchestrates these tasks, manages dependencies, and maintains a consistent state using the Delta Lake transaction log.

Key components

• Pipeline: A collection of one or more DLT tables and views defined in Python or SQL.
• Pipeline configuration: Settings like cluster size, storage location, and target schema.
• Trigger: How the pipeline is initiated — manually, on a schedule, or via file arrival.
• Execution: The actual Spark jobs that run your transformations.
• Monitoring: UI dashboards and metrics that show progress, data quality, and errors.

The life of a run

When you start a pipeline, the DLT runtime:

1. Graphs the dependencies between your tables and views.
2. Creates or updates the target schema and locations in Delta Lake.
3. Launches a Spark cluster with the configured compute resources.
4. Runs each table's transformation in the correct order, respecting dependencies.
5. Writes results to Delta tables atomically, using transaction logs to ensure consistency.
6. Updates the run status and surfaces metrics for monitoring.

The whole process is declarative — you define what you want (tables), and DLT handles how to run it efficiently. This is in stark contrast to traditional scripted ETL where you manually sequence jobs.

Pro tip: Think of DLT as a managed orchestration layer on top of Spark. You focus on the transformations, not on the scheduling and cluster management.

How it works step by step

Running a pipeline can be done through the UI, CLI, or API. Here's the logical sequence that makes it all work.

Step 1: Prepare your DLT code

Before you run, ensure your code is DLT-compliant. Use the @dlt.table decorator (Python) or LIVE TABLE (SQL) to declare tables. Here's a minimal Python example:

import dlt
from pyspark.sql.functions import col

@dlt.table
async def bronze_events():
    return (
        spark.readStream.format("cloudFiles")
        .options(path="abfss://raw@datalake.dfs.core.windows.net/events", fileFormat="json")
        .load()
    )

@dlt.table
async def silver_events():
    return dlt.read_stream("bronze_events").select(
        col("event_id"), col("event_time"), col("user_id")
    )

Step 2: Configure the pipeline

In the Databricks UI, create a pipeline and provide:

• Name and storage location (e.g., dbfs:/pipelines/my_pipeline)
• Target schema (e.g., my_analytics)
• Compute settings: Use the default or specify a custom cluster policy.
• Libraries: Attach your notebook or JAR file containing the DLT code.

Step 3: Start the run

Click Start to trigger an initial run. DLT will create the tables and start ingesting and transforming data.

Step 4: Trigger subsequent runs

For production, you'll schedule runs using the configured triggers:

• Continuous: process data as it arrives (streaming).
• Triggered: run on a schedule (e.g., hourly) or manually.

Step 5: Monitor and react

Use the pipeline UI to view run status, table details, and quality metrics. Set up alerts for failures.

Hands-on walkthrough

Let's run a real DLT pipeline end-to-end. We'll use a simple Bronze→Silver transformation.

Setup

1. Create a notebook named dlt_pipeline in your workspace.
2. Paste the Python code from above.
3. In the left sidebar, go to Workflows → Delta Live Tables.
4. Click Create Pipeline.

Configure the pipeline

In the Create Pipeline screen:

• Pipeline name: sales_pipeline
• Storage location: dbfs:/pipelines/sales (leave default if unsure)
• Target schema: sales (we'll create it)
• Notebook libraries: Add your dlt_pipeline notebook.

Trigger the run

Click Start. The UI shows the pipeline graph, and you'll see tasks appearing as they execute.

Expected output

After the run, you'll see tables like bronze_events and silver_events in the sales schema. You can query them:

%sql
SELECT * FROM sales.silver_events LIMIT 10;

You should see rows of event data. The run summary shows the number of rows processed and any failures.

Full Python example with incremental loads

Here's a more complete example that demonstrates incremental processing with apply_changes (the DLT pattern for SCD):

import dlt
from pyspark.sql.functions import col, current_timestamp

@dlt.table
async def raw_orders():
    return (
        spark.readStream.format("cloudFiles")
        .options(
            path="abfss://raw@datalake.dfs.core.windows.net/orders",
            fileFormat="json",
            multiLine=True
        )
        .load()
    )

@dlt.table
async def clean_orders():
    return dlt.read_stream("raw_orders").select(
        col("order_id"),
        col("customer_id"),
        col("order_status"),
        col("order_timestamp"),
        current_timestamp().alias("processed_at")
    ).filter(col("order_status").isNotNull())

@dlt.table
async def orders():
    return dlt.read("clean_orders")

# Apply changes for upserts (SCD type 2)
dlt.create_streaming_table("orders_final")
dlt.apply_changes(
    target="orders_final",
    source="clean_orders",
    keys=["order_id"],
    sequence_by="order_timestamp",
    apply_as_deletes=col("order_status") == "CANCELLED",
    except_column_list=["order_status"]
)

To run this pipeline:

1. Create a new pipeline in the UI.
2. Attach this notebook.
3. Set the target schema to retail.
4. Start the run and wait for it to complete.

Output will appear in the Events tab: each table shows rows read, written, and any errors. You'll see orders_final with only the latest versions of each order.

Compare options / when to choose what

Delta Live Tables pipelines don't exist in a vacuum. You have multiple ways to run and schedule your DLT code. Here's how they compare:

When choosing, consider:

• Development: Use the UI for iterative testing.
• CI/CD: Use the CLI or API to run a validation pipeline on merge requests.
• Production: Use scheduled triggers with alerting.

Pro tip: For production, always enable continuous mode for streaming pipelines so data flows in near-real-time. For batch workloads, a scheduled trigger is sufficient.

Troubleshooting & edge cases

Running a pipeline isn't always smooth. Here are common issues and how to fix them.

Issue 1: Pipeline fails with "Cannot resolve column"

Cause: Inconsistent schema between your source data and expected columns.

Fix: Use select explicitly and add expected datatype casting. Debug by inspecting the data with display() before writing.

Issue 2: Pipeline runs but tables are empty

Cause: Auto Loader didn't detect files (missing file path or wrong format).

Fix: Check the Auto Loader options—the fileFormat and path must be correct. Also ensure the storage credentials have read access.

Issue 3: Slow performance or OOM errors

Cause: Not enough cluster resources for the data volume.

Fix: Increase cluster size or implement skew handling. Use repartition where appropriate.

Issue 4: Schema drift causing failures

Cause: New fields appear in you source data.

Fix: Use schemaHints or enable cloudFiles.schemaLocation to save the inferred schema. For tolerance, you can set cloudFiles.schemaEvolutionMode to "rescue".

Issue 5: Concurrent runs conflicting

Cause: Multiple runs triggered simultaneously on the same pipeline.

Fix: DLT prevents concurrent runs by default, but if you see conflicts, check your trigger settings. Use a single trigger or lock with a queue.

Edge case: Initial run from scratch

When you run a pipeline for the first time, all tables are created. If a later stage fails, earlier tables are still valid. Subsequent runs will only process new data, thanks to Delta's transaction log.

Important: Always test with a small sample of data before a full production run. This catches schema errors early.

What you learned & what's next

You now understand how to run a delta live tables pipeline in Databricks. You know the core concepts: pipeline configuration, triggering, monitoring, and troubleshooting. You've completed a hands-on exercise and can compare different run methods.

You've met the learning objectives: - You can explain the core idea behind running a DLT pipeline—the declarative approach and lifecycle. - You can complete a practical exercise, from starting the pipeline to verifying the output.

What's next? In the next lesson, you'll learn how to optimize Delta Lake tables—a critical skill to keep your DLT pipelines fast and cost-efficient. You'll apply Z-order indexing and vacuuming to maintain performance as your data grows.

Keep your pipeline running smoothly, and happy data engineering!