Check Null Rate Threshold in PySpark DataFrame

This PySpark code checks the null rate of specified DataFrame columns against a threshold and returns violations.

Medium Python 3.8+ Aug 9, 2026 Data pipelines & processing 13 views 0 copies

pyspark data quality null check validation
Requires third-party packages — install first
pip install pyspark

Python code

39 lines
Python 3.8+ 
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, count

def check_null_rate(df, threshold=0.2, columns=None):
    """
    Check null rate for specified columns (or all) against a threshold.
    Returns columns that exceed the threshold.
    """
    cols = columns or df.columns
    total_rows = df.count()
    violations = []
    for column in cols:
        null_count = df.filter(col(column).isNull()).count()
        null_rate = null_count / total_rows
        if null_rate > threshold:
            violations.append((column, null_rate))
    return violations

if __name__ == "__main__":
    spark = SparkSession.builder.master("local[*]").appName("null-check").getOrCreate()
    
    data = [
        ("Alice", 25, "NYC"),
        ("Bob", None, "LA"),
        (None, 30, "SF"),
        ("Diana", 22, None),
        ("Eve", 35, "NYC"),
        (None, None, "LA")
    ]
    df = spark.createDataFrame(data, ["name", "age", "city"])
    
    result = check_null_rate(df, threshold=0.2)
    if result:
        for col_name, rate in result:
            print(f"FAIL: {col_name} null rate = {rate:.2%} (threshold {0.2:.0%})")
    else:
        print("PASS: all columns are within the null rate threshold")
    
    spark.stop()

Output

stdout 
FAIL: age null rate = 50.00% (threshold 20%)
FAIL: name null rate = 33.33% (threshold 20%)
FAIL: city null rate = 16.67% (threshold 20%)

How it works

The function check_null_rate computes total rows once and then counts nulls per column using isNull().count(). It returns a list of (column, null_rate) tuples for columns exceeding the threshold. The example DataFrame has 6 rows with nulls in name, age, and city, producing null rates above the 20% threshold. This pattern is common in data quality validation to alert on columns with excessive missing data.

Common mistakes

  • Forgetting to divide by total rows and comparing raw null counts
  • Not handling an empty DataFrame causing division by zero
  • Using `dropna` or `fillna` instead of counting nulls accurately
  • Hardcoding column names instead of using the function's `columns` parameter

Variations

  1. Use an aggregation with `sum(col(column).isNull().cast('int'))` for a single pass
  2. Use `df.select([(count(col(c)) / count('*')).alias(c) for c in df.columns])` to compute all null rates

Real-world use cases

  • Validating data quality in an ETL pipeline before loading into a warehouse
  • Triggering alerts when source data has unexpected null spikes
  • Comparing null rates across time windows to detect data drift

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from Data pipelines & processing

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes