Filter Records by Required Fields in Python

Filter a list of dictionaries, keeping only records where every required field is present and not None.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

filter data-cleaning pipelines dictionaries

Python code

18 lines

Python 3.9+

def filter_records(records, required_fields):
    """Return only records that have all required fields non-null."""
    return [
        record for record in records
        if all(record.get(field) is not None for field in required_fields)
    ]


if __name__ == "__main__":
    sample_records = [
        {"name": "Alice", "email": "alice@example.com", "age": 30},
        {"name": "Bob", "email": None, "age": 25},
        {"name": "Carol", "email": "carol@example.com", "age": None},
        {"name": "Dave", "email": "dave@example.com", "age": 40},
    ]
    required = ["name", "email", "age"]
    filtered = filter_records(sample_records, required)
    print(filtered)

Output

stdout

[{'name': 'Alice', 'email': 'alice@example.com', 'age': 30}, {'name': 'Dave', 'email': 'dave@example.com', 'age': 40}]

How it works

The all() function checks that every field in required_fields evaluates to a truthy condition for the record. Using record.get(field) is not None ensures the field exists (or defaults to None) and is not explicitly set to None. This creates a clean, composable filter for data pipelines where missing or null values invalidate a record.

Common mistakes

Using `record.get(field)` without checking for None, which might pass when a key is missing but returns None.
Confusing `None` with empty strings — this code treats empty strings as valid data, so adjust if needed.
Forgetting that `all()` returns True for an empty `required_fields` list, so all records pass.

Variations

Use a lambda with `filter()`: `list(filter(lambda r: all(r.get(f) is not None for f in required), records))`
Use a comprehension that also fills missing keys with a default value before filtering.

Real-world use cases

In an ETL job, drop rows with missing critical columns (like user_id or timestamp) before loading.
Validate API responses by filtering out objects that lack mandatory fields before processing.
In a data pipeline, clean event streams by retaining only records with non-null tracking identifiers.

Filter Records by Required Fields in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Data pipelines & processing

Keep learning

Tutorials

Quizzes