How to Build a Data Validation Schema in Python

Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.

Easy Python 3.9+ Aug 9, 2026 ML engineering pipelines 12 views 0 copies

Python code

48 lines
Python 3.9+
import re
from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class Field:
    name: str
    validator: Callable[[Any], bool]
    required: bool = True

    def validate(self, value: Any) -> bool:
        if not self.required and value is None:
            return True
        return self.validator(value)


@dataclass
class Schema:
    fields: list[Field] = field(default_factory=list)

    def add(self, name: str, validator: Callable[[Any], bool], required: bool = True) -> None:
        self.fields.append(Field(name, validator, required))

    def validate(self, data: dict) -> dict[str, str]:
        errors = {}
        for field_def in self.fields:
            value = data.get(field_def.name)
            if not field_def.validate(value):
                errors[field_def.name] = f"Invalid value: {value!r}"
        return errors


def is_email(value: str) -> bool:
    return bool(re.match(r"^[^@]+@[^@]+\.[^@]+$", value))


if __name__ == "__main__":
    schema = Schema()
    schema.add("email", is_email)
    schema.add("age", lambda v: isinstance(v, int) and 0 <= v <= 120, required=False)
    schema.add("name", lambda v: isinstance(v, str) and len(v) > 0)

    sample = {"email": "user@example.com", "age": 30, "name": "Alice"}
    print("Valid data errors:", schema.validate(sample))

    bad_sample = {"email": "not-an-email", "age": -5, "name": ""}
    print("Invalid data errors:", schema.validate(bad_sample))

Output

stdout
Valid data errors: {}
Invalid data errors: {'email': "Invalid value: 'not-an-email'", 'age': 'Invalid value: -5', 'name': "Invalid value: ''"}

How it works

The Field dataclass bundles a validator callable with a required flag, and its validate method skips validation when a non-required field is missing or None. The Schema class collects these fields and iterates over them on each validate call, building an error dictionary keyed by field name. Using f"Invalid value: {value!r}" gives a readable repr of the rejected value. Optional fields are handled by setting required=False, so missing None values pass. This pattern mirrors how production pipelines or ML preprocessing steps check incoming data before further processing.

Common mistakes

  • Forgetting to pass a default value for `required` when calling `add` and accidentally making optional fields required.
  • Using a lambda that raises an exception (e.g., `len` on non-string) instead of returning a boolean, causing validation to crash.
  • Not handling `None` for optional fields; here the `Field.validate` explicitly returns `True` if the field is not required and value is `None`.
  • Assuming `data.get()` returns the value for a missing key; it returns `None`, which may be incorrectly flagged as invalid for required fields.

Variations

  1. Use Pydantic's `BaseModel` with type hints and custom validators for a more robust, well-known schema library.
  2. Add a `coerce` function to transform values before validation (e.g., casting string numbers to int).

Real-world use cases

  • Validating feature vectors or metadata before feeding them into a model training pipeline.
  • Checking API request payloads against required and optional fields in a microservice handler.
  • Sanitizing and verifying configuration dictionaries loaded from environment or files in ML experiment runners.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from ML engineering pipelines

Related tutorials and quizzes for this topic.