Create Your First Delta Table

You've built DataFrames, run transformations, and queried data in notebooks — but every time you restart a cluster, that work vanishes. Your data pipeline currently ends in a dead end: results exist only in memory, not as a durable, queryable asset your team can trust. The solution is Delta Lake, and the first step is learning how to create your first Delta table — the foundation of every reliable data pipeline on Databricks. In this lesson, you'll move from ephemeral DataFrames to persistent, versioned tables that support ACID transactions, time travel, and schema evolution.

The problem this lesson solves

Imagine you've spent hours cleaning and transforming a dataset, but when you try to share your results with a colleague, they can't see them. Or worse, the next day your pipeline fails because an upstream file changed format without notice. These are classic symptoms of working with raw files or in-memory data without a governed storage layer.

Raw Parquet or CSV files on cloud storage have no transaction support, no schema enforcement, and no versioning. If two jobs write to the same directory concurrently, you can corrupt your data. Delta Lake solves these problems by adding a transaction log and ACID guarantees on top of your existing storage. Creating your first Delta table is the single action that turns a collection of files into a reliable, queryable dataset.

Think of a Delta table as a database table that lives on your cloud storage — it's not a separate service, just a directory with a special _delta_log folder that tracks every change.

Core concept / mental model

The fastest way to grasp Delta tables is by analogy: a Delta table is like a bank ledger. Every transaction (insert, update, delete) is recorded in a log before the data changes. This log ensures that all reads see a consistent snapshot, even if multiple writers are active. If something goes wrong, you can roll back to a previous state — exactly like reviewing a bank statement.

Here are the key components you'll work with:

• Table location: A directory on cloud storage (e.g., dbfs:/mnt/delta/events).
• Data files: Parquet files that store the actual rows.
• Transaction log: A _delta_log subdirectory containing JSON files — each file is a commit that describes what changed.
• Table metadata: Schema, partitioning, and configuration stored in the log.

Delta tables support standard SQL and DataFrame operations, so you don't need to learn a new language — you just wrap your existing Spark code with the DELTA format.

How it works step by step

Creating a Delta table from a DataFrame is a three-step process:

1. Load your source data into a Spark DataFrame using any of Databricks' connectors (Parquet, CSV, JSON, JDBC, or cloud storage).
2. Write the DataFrame using .write.format("delta") and specify a path. This creates the table directory and initializes the transaction log.
3. Register the table in the metastore (optional) with CREATE TABLE ... USING DELTA or df.write.saveAsTable(), so it appears in the Data Explorer and can be queried by name.

The critical point: the first write creates the schema and the table. Every subsequent write appends or overwrites data, and each write appends a new commit to the transaction log.

Hands-on walkthrough

Let's create your first Delta table in a Databricks notebook. We'll start with a small CSV dataset and work through the standard patterns.

1. Create a Delta table from a CSV

First, load a sample CSV from the Databricks datasets:

# Load sample data
df = spark.read.csv("/databricks-datasets/COVID/covid-19-data/us-counties.csv",
                    header=True, inferSchema=True)

# Write to a Delta table (this is step one — your first Delta table!)
df.write.format("delta").mode("overwrite").save("/mnt/delta/covid_us")  # Note: use DBFS root path in notebooks

If you run this, you'll see a message like Delta Lake updated 5 files in 0.89 seconds. Check the location with:

display(spark.sql("DESCRIBE EXTENDED delta.`/mnt/delta/covid_us`"))

You'll see the table schema, location, and format — all confirmed as Delta.

2. Register the table in the metastore

To make the table accessible by name across your workspace, register it:

# Register as a managed table in the default database
spark.sql("CREATE TABLE covid_us USING DELTA LOCATION '/mnt/delta/covid_us'")
# Or with saveAsTable (creates a managed table in the database)
df.write.saveAsTable("covid_us")

Now you can query it with SQL:

SELECT state, COUNT(*) FROM covid_us GROUP BY state ORDER BY COUNT(*) DESC LIMIT 10;

3. Append new data and verify time travel

Delta tables shine with updates. Let's append a few fake rows and then use time travel to see the original version:

# Append new data
from pyspark.sql import Row
new_rows = spark.createDataFrame([
    Row(date="2021-01-01", county="Test", state="XX", fips=999, cases=0, deaths=0)
])
new_rows.write.format("delta").mode("append").save("/mnt/delta/covid_us")

# Show table history
spark.sql("HISTORY covid_us").show(5, truncate=False)

# Query version 0 (original)
spark.sql("SELECT COUNT(*) FROM covid_us VERSION AS OF 0").show()

You should see counts change, and the history shows two versions.

Compare options / when to choose what

When creating a Delta table, you have several choices — especially around table type and write modes. Here's a quick guide:

For most ETL pipelines, start with external tables if data already lives in cloud storage, otherwise use managed tables. Use append for log-style data and merge for transactional changes.

Troubleshooting & edge cases

"Path does not exist" when reading a Delta table

Ensure the path is correct and permissions allow access. In Databricks, use dbfs:/ for DBFS paths, or the direct cloud path (e.g., s3://). Check with dbutils.fs.ls().

Schema mismatch when appending

If the new DataFrame has a different column order or data type, the write may fail. Use .option("mergeSchema", "true") to allow schema evolution:

df_new.write.format("delta").option("mergeSchema", "true").mode("append").save("/mnt/delta/covid_us")

"Cannot resolve 'column'" in SQL after saving as table

Your query uses a column that doesn't exist — check the schema with DESCRIBE covid_us.

Table appears empty after overwrite

Did you use mode("overwrite")? It replaces the entire table — if you intended to update a few rows, use MERGE.

Overwriting a table while other jobs are reading

Delta's optimistic concurrency control will prevent lost updates — you may get a conflict error. Retry the operation or use MERGE to avoid conflicts.

What you learned & what's next

You've created your first Delta table, registered it, appended data, and used time travel — that's the ACID foundation. You can explain how a Delta table stores data in Parquet plus a transaction log, and you can confidently choose between managed/external tables and write modes.

Next up: In the next lesson, you'll learn to read and transform data from your Delta table — including how to handle schema evolution and optimize performance with Z-order and partitioning. Now that you have a durable asset, you can build streaming and batch pipelines that everyone can trust.

Practice saving the same DataFrame to managed and external tables, then drop one and observe the behavior — it's the fastest way to internalize the difference.