Declare Expectations for Data Quality

Your pipeline ran last night, your dashboards look green, and then your CFO asks for headcount trends — only to discover that 4% of the rows in the sales table have NULL revenue and another 2% reference deleted customers. Nobody declared expectations for data quality checks, so bad data slipped through silently. In this lesson, you'll learn how to declare expectations for data quality checks on Databricks — turning vague hopes like "the data should be clean" into precise, enforceable rules that fail fast, alert you, and keep your Lakehouse trustworthy.

The problem this lesson solves

Without declared expectations, data quality is reactive. You only discover problems after downstream reports are wrong, models degrade, or users file complaints. The core pain points you'll eliminate by declaring expectations for data quality checks include:

• Silent corruption — invalid values, missing keys, or unexpected schemas pass through unnoticed.
• No single source of truth — every notebook has its own ad-hoc if-else checks, duplicated and inconsistent.
• Slow debugging — you spend hours tracing which step introduced the bad data, instead of catching it at the source.

Declaring expectations means writing explicit, machine-readable rules that define what "good data" looks like before you process it. You get immediate feedback when data violates those rules — and you can decide whether to fail, warn, or drop the bad rows. It turns quality from a post-mortem into a gate.

Pro tip: In Databricks, the quickest way to declare expectations is with Delta Live Tables (DLT) and its expect clause. But you can also use notebook-based checks or run constraints directly against Delta tables. We'll compare these options in Compare options.

Core concept / mental model

Think of declaring expectations like writing a contract between your data producers and consumers. A contract says: "I promise this field will never be null, that this column always contains a positive number, and that this key always exists in the dimension table." The data pipeline enforces the contract.

In Databricks, the three core building blocks are:

• Expectation — a named rule that checks a condition on your data (e.g., sales_revenue_not_null).
• Constraint — a declarative rule applied to a table (e.g., NOT NULL on a column).
• Validation mode — how you react when a rule fails (warn, drop, or fail).

Definitions you'll use

The pipeline as a gatekeeper

Imagine your data pipeline as a factory assembly line. Declaring expectations is like installing quality-check stations at every conveyor belt. When a part (row) fails inspection, you can either flag it (warn), reject it (drop), or stop the line (fail). This mental model helps you decide where to put checks: at the source, after each transformation, or at the final sink.

How it works step by step

Here's the logical sequence you'll follow to declare expectations for data quality checks:

1. Define your quality rules — based on business requirements and known data patterns. Write them as SQL-like conditions (e.g., col IS NOT NULL, col > 0, col IN (...)).
2. Choose your enforcement strategy — decide whether to warn, drop, or fail on violation. This depends on how critical the rule is.
3. Implement the expectations — write a DLT pipeline (recommended) or use delta_table constraints.
4. Run the pipeline — execute the job and inspect the data quality metrics that Databricks automatically tracks (number of rows that passed, failed, etc.).
5. Tune and iterate — monitor your expectations in production, adjust thresholds, and add new rules as data evolves.

Cause and effect: a precise expectation causes an immediate alert when data is bad; without it, you only see the effect days later when a report is wrong.

Hands-on walkthrough

Example 1: Declare expectations in a DLT pipeline

The cleanest way to declare expectations for data quality checks in Databricks is using Delta Live Tables. Here's a complete example:

import dlt
from pyspark.sql.functions import col, expr

# Define a table with expectations
@dlt.table(
    comment="Raw sales data with quality checks"
)
@dlt.expect_all({
    "valid_revenue": "revenue > 0",
    "customer_id_not_null": "customer_id IS NOT NULL",
    "valid_country": "country IN ('US', 'CA', 'MX')"
})
def sales_raw():
    return (
        spark.readStream.format("cloudFiles")
          .option("cloudFiles.format", "csv")
          .load("/mnt/landing/sales")
          .select("transaction_id", "customer_id","revenue", "country")
    )

When you run this DLT pipeline, Databricks will check every row against these three rules. By default, violations are dropped (you can change the action). The UI shows a quality metrics panel with counts of rows that passed and failed.

Expected output: The pipeline completes, and the DLT dashboard shows 3 expectations with pass counts. Rows that violate any rule are excluded from the target table.

Example 2: Using expect_all with drop vs. fail

You can control what happens on violation with the onviolation parameter:

@dlt.expect_all_or_fail({
    "valid_revenue": "revenue > 0",
    "customer_id_not_null": "customer_id IS NOT NULL"
})
@dlt.table
def sales_validated():
    ...

With expect_all_or_fail, any violation fails the pipeline — this is great for critical rules where you must stop processing. Use expect_all_or_drop (the default) for filtering bad rows, and expect_all for warning only (rows are kept, but you add a _warnings column).

Example 3: Declaring constraints on a Delta table

You can also declare expectations directly on a Delta table using ALTER TABLE ... ADD CONSTRAINT:

ALTER TABLE sales ADD CONSTRAINT valid_revenue CHECK (revenue > 0);
ALTER TABLE sales ADD CONSTRAINT customer_id_not_null CHECK (customer_id IS NOT NULL);

Then, any write that violates these constraints fails. This is a lower-level alternative when you're not using DLT, but it's less flexible for streaming or incremental pipelines.

Example 4: Checking quality in a notebook

Finally, you can write your own checks using PySpark in a notebook — useful for quick ad-hoc validation before publishing data:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
sales_df = spark.read.table("sales")

# Count violations
violations = sales_df.filter("revenue <= 0 OR customer_id IS NULL")
print(f"Violations: {violations.count()}")

# Assert - fail fast if too many
assert violations.count() < 100, "Too many quality violations!"

This violates the "single source of truth" principle, but can be a pragmatic stopgap.

Compare options / when to choose what

When to choose what:

• Use DLT expectations when you're building a multi-step pipeline and want per-step quality gates with automatic metrics.
• Use Delta constraints when you have a single table and want lightweight schema-level guarantees (e.g., non-nullable columns).
• Use notebook checks for exploration and pre-validation, but avoid relying on them in production due to duplication.

Variations

• Great Expectations — an open-source Python library that integrates with Databricks notebooks for richer, JSON-based expectations. It's great for sharing quality suites across teams.
• dbt tests — if you're using dbt for transformations, you can write not_null, unique, and custom tests in SQL — a different paradigm but same goal.
• System tables — Databricks also provides system tables for monitoring pipeline health and data quality metrics over time.

Troubleshooting & edge cases

Common issue 1: NULL values still appear in the table

• Symptom: You declared customer_id IS NOT NULL, but run a query and see NULLs.
• Cause: You used expect_all (warning mode) instead of expect_all_or_drop or expect_all_or_fail. Warnings keep the rows.
• Fix: Switch to a stricter mode or check the _warnings column.

Common issue 2: Pipeline fails unexpectedly on low-volume days

• Symptom: A fail expectation triggers when only a few rows are bad, breaking the whole job.
• Cause: Your rule is too strict for normal variation.
• Fix: Use a warning first, monitor the metrics, then adjust the threshold — or use a percentage-based check via SQL (e.g., count(*) <= 0.01 * total).

Common issue 3: Constraints don't apply to existing rows

• Symptom: You add a CHECK constraint but historical bad rows remain.
• Cause: ADD CONSTRAINT only validates new writes; existing rows aren't retroactively checked.
• Fix: Run a separate validation query on the existing table and clean the data before adding the constraint.

Common issue 4: Not sure if a check ran

• Symptom: No quality metrics visible in the UI.
• Cause: You may have used a notebook-based check, which doesn't produce DLT metrics.
• Fix: Switch to DLT expectations to get automatic metrics — or at least log your assertions in a structured way.

Pro tip: Always start with warn for a few runs. Look at the metrics, see how many rows fail, then decide whether to drop or fail. This prevents surprises in production.

What you learned & what's next

You now understand the core idea behind declaring expectations for data quality checks: you moved from silent bad data to explicit, enforceable contracts. You practiced writing expectations using Delta Live Tables, applying Delta constraints, and doing ad-hoc notebook checks. You can now:

• Explain what it means to declare expectations for data quality checks in Databricks.
• Complete a practical exercise to define and run quality checks on your own tables.

Your key takeaways:

• Declaring expectations makes quality part of the pipeline, not an afterthought.
• DLT provides the richest syntax with expect, expect_all, expect_all_or_drop, and expect_all_or_fail.
• Delta constraints are a simple, low-cost alternative for batch tables.
• You choose between warn, drop, and fail based on business criticality.
• Monitor quality metrics in the DLT UI to tune your rules.

Next lesson: Now that your data satisfies expectations, you're ready to enforce data quality with constraints — diving deeper into schema enforcement, nullability, and advanced CHECK constraints. That lesson builds directly on the expectations you declared here.

Practice recap: As a mini-exercise, rewrite one of your existing ETL notebooks to add a DLT expect_all on the final table where you know the business rules. Run it with warn first, check the metrics, then switch to expect_all_or_drop for anything that should never reach the next step.