Delta Lake Table Basics

You've mastered notebooks, clusters, and Spark DataFrames — but when you write that DataFrame to a table, do you actually trust the data that comes back out? Without proper table management, you can end up with corrupted files, inconsistent reads, or silent data loss. That's exactly the pain Delta Lake solves. In this lesson, you'll understand Delta Lake table basics — what they are, how they work, and how to manage them with confidence on Databricks.

The problem this lesson solves

Traditional Parquet tables on cloud storage are fragile. If a job fails mid-write, you're left with partial data. If two processes write to the same location simultaneously, they could overwrite each other. And if you try to upsert or delete records, you end up rewriting entire files — slow and error-prone.

Without a transactional layer, you also have no time travel — you can't see what the data looked like yesterday. And schema changes? One bad column rename can break downstream analytics for weeks.

Delta Lake is a storage layer that brings ACID transactions, scalable metadata handling, and data versioning to your existing Parquet files. Once you grasp the core table basics, you'll stop fighting your storage and start trusting it.

Core concept / mental model

Think of a Delta table as a cloud-native database disguised as a file folder. Under the hood, it stores data in Parquet format, but it adds a critical _delta_log directory containing transaction records.

What is a Delta table?

A Delta table is a collection of Parquet data files plus a transaction log (the _delta_log folder). Every change you make — insert, update, delete, merge, or schema change — is recorded as an atomic entry in that log.

Key components: - Data files: Parquet files that store columns and rows. - Transaction log: JSON files that list data files added, removed, and metadata changes. - Checkpoint files: Parquet snapshots that speed up log reads.

Table format vs. external data source

In Databricks, a Delta table can be managed (stored in the Databricks-managed location) or external (pointing to a location you control, like an S3 bucket). Managed tables are simpler; external tables give you flexibility.

Mental model: Think of the transaction log as a receipt of every operation. When you read the table, Spark reads the latest receipt to know which files are part of the current version. Everything else is just Parquet.

How it works step by step

Understanding Delta Lake table basics means knowing how the engine processes your operations and reads data back. Let's trace the lifecycle of a table.

1. Creating a table

When you write a DataFrame with .format("delta").save() or use SQL CREATE TABLE, Databricks creates the _delta_log directory and writes an initial transaction entry.

2. Writing data

Every write (append, overwrite, upsert) creates new Parquet files and records a commit in the log. If a commit fails, the previous version remains intact — no partial writes.

3. Reading data

The reader fetches the latest log, identifies active files, and reads them. Because the log is the source of truth, reads always see a consistent snapshot.

4. Time travel

You can query any previous version using VERSION AS OF syntax. This is possible because the log maintains history.

5. Vacuuming

Old data files are cleaned up periodically with VACUUM to remove files no longer referenced by the log.

Hands-on walkthrough

Let's get practical. Open a Databricks notebook and follow along. We'll create a Delta table, perform operations, and explore its internals.

Step 1: Create a DataFrame and write as Delta

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Create a simple DataFrame
spark = SparkSession.builder.appName("DeltaBasics").getOrCreate()
data = [("Alice", 28, "Sales"), ("Bob", 34, "Eng")]
df = spark.createDataFrame(data, ["name", "age", "dept"])

# Write as Delta (managed table)
df.write.format("delta").saveAsTable("people")

After running this, you'll see a new table people in the Databricks catalog. If you want an external location:

df.write.format("delta").save("/mnt/external/path/people_delta")

Step 2: Read and check history

# Read the table back
people_df = spark.table("people")
people_df.show()

# Check transaction history
%sql
DESCRIBE HISTORY people;

Expected output (simplified):

+-------+---+----+
|   name|age|dept|
+-------+---+----+
|  Alice| 28|Sales|
|    Bob| 34|  Eng|
+-------+---+----+

Step 3: Upsert with MERGE

Delta Lake supports MERGE, which is critical for slowly changing dimensions (SCD). Let's add a new person and update an existing one:

new_data = [("Bob", 35, "Eng"), ("Carol", 29, "Finance")]
new_df = spark.createDataFrame(new_data, ["name", "age", "dept"])

# Merge new_df into people based on name
from delta.tables import DeltaTable
people_delta = DeltaTable.forName(spark, "people")
people_delta.alias("p").merge(
    new_df.alias("n"),
    "p.name = n.name") \
  .whenMatchedUpdate(set={
      "age": "n.age",
      "dept": "n.dept"
  }) \
  .whenNotMatchedInsertAll() \
  .execute()

spark.table("people").show()

Output:

+-----+---+-------+
| name|age|   dept|
+-----+---+-------+
| Alice| 28|  Sales|
|   Bob| 35|    Eng|
| Carol| 29|Finance|
+-----+---+-------+

Step 4: Time travel

# Show current version
%sql
SELECT * FROM people VERSION AS OF 1;

Compare options / when to choose what

Delta Lake is not the only choice on Databricks. Let's compare it with common alternatives.

When to use Delta Lake

• Data pipelines that need reliability
• Streaming + batch consolidation (Structure Streaming writes directly)
• Regulatory requirements needing audit history

When not to use Delta Lake

• Quick prototyping with tiny datasets? Plain Parquet might suffice.
• If you need external tools that don't support Delta format — though today compatibility is broad.

Troubleshooting & edge cases

Error: AnalysisException: A MERGE statement ...

Cause: The merge condition referenced a column not in the target or source.

Fix: Verify column names and use aliases consistently.

Wrong output after adding a column

Cause: Schema evolution is disabled by default.

Fix: Enable it with spark.databricks.delta.schema.autoMerge.enabled=true or use ALTER TABLE ADD COLUMNS.

Time travel fails with Cannot time travel to version X

Cause: The file was vacuumed.

Fix: Increase vacuum retention or avoid vacuuming on critical history.

Large transaction log slowing reads

Issue: Too many small entries.

Solution: Run OPTIMIZE to compact files and VACUUM to clean old snapshots.

What you learned & what's next

You now know the core Delta Lake table basics: the structure (_delta_log), how to create, read, upsert, and time-travel. You can compare Delta with other formats and troubleshoot common issues. You're ready to apply this to real ETL pipelines.

Next, you'll learn how to optimize Delta tables with OPTIMIZE and ZORDER to boost query performance. Keep this foundation solid, and you'll be building production-grade lakehouse solutions in no time.