Create a Live Table with Delta Live Tables

You've built batch pipelines that work — until the schema changes, the data lands late, or your boss asks for a dashboard that reflects the last five minutes. Manually orchestrating Spark jobs and praying the data is fresh is a recipe for late nights. Delta Live Tables (DLT) on Databricks removes the pain: you declare your datasets as live tables, and the platform builds, runs, and maintains the entire pipeline for you. In this lesson, you'll learn how to create a live table with Delta Live Tables, run a pipeline, and adopt best practices that turn brittle ETL into a self-healing, production-grade system.

The problem this lesson solves

Classic ETL pipelines are fragile. You write a notebook, schedule a job, and hope the upstream data is there. When something breaks, you debug Spark logs at 2 a.m. And even when it works, your data can be stale, duplicated, or missing quality checks.

Delta Live Tables solves these problems by introducing a declarative model: you define what each table should be, not how to build it. A live table is a dataset that DLT manages automatically — it tracks dependencies, computes the data in the right order, and refreshes incrementally when new data arrives. Think of DLT as a pipeline-as-code layer on top of Delta Lake: it handles orchestration, retries, and monitoring so you can focus on the logic.

The specific pain points DLT addresses:

• Orchestration complexity — no more chaining notebooks or managing Airflow DAGs.
• Schema drift — DLT can enforce, evolve, or quarantine schema changes automatically.
• Data quality — built-in constraints and expectation checks fail fast or quarantine bad rows.
• Incremental processing — you get streaming and batch in one unified API; no separate code paths.
• Infrastructure management — DLT provisions and scales the underlying compute for you.

Pro tip: If you've ever been burned by a silent schema change breaking a downstream table, DLT's schema evolution and expectations will feel like a superpower.

Core concept / mental model

Imagine you're building a house. A traditional pipeline is like hand-carrying each brick, mixing cement, and checking the walls yourself every time. DLT is like having a general contractor: you hand them a blueprint (your Python or SQL code), and they manage the crew, the materials, and the inspections. You only need to review the final structure.

In DLT, the blueprint is a set of live table declarations — functions or SQL statements that define a dataset. Each live table is a Node in a directed acyclic graph (DAG). DLT reads your code, builds the DAG, then executes it in the correct order, parallelizing independent branches.

Key terms:

• Live table — a dataset managed by DLT; can be a view, a materialized view, or a streaming table.
• Pipeline — the execution unit that runs all your live tables, with associated compute and configuration.
• Expectations — data quality rules (like constraints) that DLT enforces.
• Target schema — the database where DLT writes your live tables (e.g., main.default).

Here's a mental picture of a DLT pipeline:

[Cloud Files / Kafka] → Bronze (raw, streaming) → Silver (clean, deduplicated) → Gold (aggregated)

Each stage is a live table. DLT manages the flow, and you only write the definitions.

How it works step by step

Creating a live table with Delta Live Tables follows a clear pattern — you define, then run. Here's the high-level flow:

1. Author DLT code — in a notebook or file, import dlt and declare at least one @dlt.table decorated function (or use CREATE LIVE TABLE in SQL).
2. Create a pipeline — in the Databricks UI, click Delta Live Tables → Create Pipeline, name it, and attach your notebook or file path.
3. Configure compute and target — choose a serverless or classic compute profile, and set the target schema (e.g., main.default).
4. Run the pipeline — start a triggered or continuous run. DLT builds the DAG and executes all tables.
5. Monitor and iterate — check the pipeline UI for metrics, data quality, and lineage. Adjust code and rerun.

The key is that DLT tracks metadata about each table: the version, the data, and the dependencies. When you rerun, DLT only recomputes what changed, using incremental processing where possible.

Declarative vs. imperative

In a traditional notebook, you write imperative code: df = spark.read... and df.write.... With DLT, you write declarative code: `@dlt.table` tells DLT, "this is a table; here's its definition." DLT decides when and how to compute it.

The role of expectations

You can attach expectations to a live table to enforce data quality. For example, you can require that id is not null, or that amount is positive. If a row violates an expectation, you can warn, drop, or fail the pipeline. This is a game-changer for production pipelines.

Hands-on walkthrough

Let's get practical. In this example, we'll create a simple DLT pipeline with two live tables: a raw (bronze) table and a cleaned (silver) table. We'll use Python — the same pattern applies to SQL.

Step 1: Create a notebook and set up your code

In your Databricks workspace, create a new notebook (use a cluster-independent Python notebook — DLT manages the cluster). Paste the following code in a cell:

import dlt
from pyspark.sql.functions import col, current_timestamp

# Define a live table that reads from a file path (e.g., a CSV landing zone)
@dlt.table(
    comment="Raw sales data from external system",
    table_properties={
        "quality": "bronze"
    }
)
def raw_sales():
    return (
        spark.read.format("cloudFiles")
        .option("cloudFiles.format", "csv")
        .load("/databricks-datasets/retail-org/sales_orders_csv")
    )

# Define a live table that cleans the raw data
@dlt.table(
    comment="Cleaned sales data with data quality checks",
    table_properties={
        "quality": "silver"
    }
)
@dlt.expect("valid_quantity", "quantity > 0")
@dlt.expect_or_drop("valid_customer_id", "customer_id IS NOT NULL")
def cleaned_sales():
    return (
        dlt.read("raw_sales")
        .select(
            col("order_id"),
            col("customer_id"),
            col("amount"),
            col("quantity"),
            current_timestamp().alias("ingested_at")
        )
    )

Step 2: Create and run the pipeline

• In the sidebar, click Delta Live Tables.
• Click Create Pipeline.
• Name it sales_pipeline.
• In Notebook Libraries, add your notebook.
• In Target schema, enter main.default (or a database you have access to).
• For Compute, choose the Serverless option (if available) or leave defaults.
• Click Create and then Start to run the pipeline.

Expected output: The pipeline will show two tables, raw_sales and cleaned_sales, with Created or Updated status. You'll see metrics like rows written and data quality pass/fail counts.

Step 3: Query your live tables

After the run, go to the SQL Warehouse or a SQL notebook and run:

SELECT * FROM main.default.cleaned_sales;

You'll see the cleaned data with the ingested_at timestamp. The raw_sales table is still there for lineage and debugging.

Step 4: Modify and rerun (CI/CD ready)

Change a business rule — for example, add a new column transformation — then rerun the pipeline. DLT will recompute only the affected tables and preserve downstream data if possible. This is the power of declarative pipelines.

Compare options / when to choose what

You might wonder: when should I use DLT versus a classic notebook job or Spark structured streaming? Here's a quick comparison:

When to choose DLT:

• You have multiple tables with dependencies.
• You need incremental refresh on a schedule.
• You want built-in data quality to prevent bad data from reaching analysts.
• You want to reduce orchestration overhead.

When to avoid DLT:

• You need to integrate with a non-Databricks orchestrator for cross-platform dependencies.
• You're doing heavy custom Spark tuning that the DLT abstraction might hide.
• You only have a one-off query — a simple notebook is fine.

Variations

• SQL live tables — you can use CREATE LIVE TABLE and CREATE STREAMING LIVE TABLE in SQL notebooks, which is great for SQL-savvy teams.
• Views vs. materialized views — DLT offers @dlt.view (always recomputed on refresh) vs. @dlt.table (persisted). Choosing depends on whether you need to persist intermediate results.
• Auto Loader vs. spark.read — for incremental file ingestion, Auto Loader (via .format("cloudFiles")) is the recommended option, as it can detect new files without from_timestamp tricks.

Troubleshooting & edge cases

DLT is robust, but you will hit issues. Here are common ones and how to fix them:

• Pipeline fails with 'Table not found' — Check your dependencies. Did you use dlt.read("table_name") with the correct name? DLT uses the function name or the name parameter in the decorator.
• Schema mismatch on a re-run — If you change a column's type, DLT might fail or need manual refresh. Use schema evolution settings in the pipeline, or handle with @dlt.expect_or_drop to filter bad rows.
• Incremental update returns no new data — For streaming tables, ensure your source is actually receiving new events. For Auto Loader, verify that new files are being written to the ingestion path.
• Expectation violations with expect_or_drop — Rows are silently dropped. Check the metrics to see if you're losing too much data; adjust the expectation to expect (warn) first.
• Compute failure due to out-of-memory — Increase the cluster size or use Serverless compute. DLT's autoscaling helps, but check your spark.sql.shuffle.partitions for large shuffles.

Pro tip: Always run a pipeline with a CREATE (full refresh) first using a small sample dataset to validate logic before enabling incremental updates.

What you learned & what's next

You've learned how to create a live table with Delta Live Tables: you understand the declarative model, you've written a Python DLT pipeline with expectations, and you've run it to produce live tables in your target schema. You also know when DLT beats classic pipelines and how to troubleshoot common pitfalls.

Now that you can define live tables, the next logical step is to manage table dependencies and add more sophisticated data quality checks — expect to build multi-stage bronze/silver/gold pipelines with incremental refreshes and expectation policies that keep your Lakehouse trustworthy.

Keep experimenting: try adding a third table, using Auto Loader with JSON files, or setting up a continuous pipeline for real-time tables.