How to Build a Data Validation Schema in Python
Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.
Python code
48 linesimport re
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Field:
name: str
validator: Callable[[Any], bool]
required: bool = True
def validate(self, value: Any) -> bool:
if not self.required and value is None:
return True
return self.validator(value)
@dataclass
class Schema:
fields: list[Field] = field(default_factory=list)
def add(self, name: str, validator: Callable[[Any], bool], required: bool = True) -> None:
self.fields.append(Field(name, validator, required))
def validate(self, data: dict) -> dict[str, str]:
errors = {}
for field_def in self.fields:
value = data.get(field_def.name)
if not field_def.validate(value):
errors[field_def.name] = f"Invalid value: {value!r}"
return errors
def is_email(value: str) -> bool:
return bool(re.match(r"^[^@]+@[^@]+\.[^@]+$", value))
if __name__ == "__main__":
schema = Schema()
schema.add("email", is_email)
schema.add("age", lambda v: isinstance(v, int) and 0 <= v <= 120, required=False)
schema.add("name", lambda v: isinstance(v, str) and len(v) > 0)
sample = {"email": "user@example.com", "age": 30, "name": "Alice"}
print("Valid data errors:", schema.validate(sample))
bad_sample = {"email": "not-an-email", "age": -5, "name": ""}
print("Invalid data errors:", schema.validate(bad_sample))
Output
Valid data errors: {}
Invalid data errors: {'email': "Invalid value: 'not-an-email'", 'age': 'Invalid value: -5', 'name': "Invalid value: ''"}
How it works
The Field dataclass bundles a validator callable with a required flag, and its validate method skips validation when a non-required field is missing or None. The Schema class collects these fields and iterates over them on each validate call, building an error dictionary keyed by field name. Using f"Invalid value: {value!r}" gives a readable repr of the rejected value. Optional fields are handled by setting required=False, so missing None values pass. This pattern mirrors how production pipelines or ML preprocessing steps check incoming data before further processing.
Common mistakes
- Forgetting to pass a default value for `required` when calling `add` and accidentally making optional fields required.
- Using a lambda that raises an exception (e.g., `len` on non-string) instead of returning a boolean, causing validation to crash.
- Not handling `None` for optional fields; here the `Field.validate` explicitly returns `True` if the field is not required and value is `None`.
- Assuming `data.get()` returns the value for a missing key; it returns `None`, which may be incorrectly flagged as invalid for required fields.
Variations
- Use Pydantic's `BaseModel` with type hints and custom validators for a more robust, well-known schema library.
- Add a `coerce` function to transform values before validation (e.g., casting string numbers to int).
Real-world use cases
- Validating feature vectors or metadata before feeding them into a model training pipeline.
- Checking API request payloads against required and optional fields in a microservice handler.
- Sanitizing and verifying configuration dictionaries loaded from environment or files in ML experiment runners.
Sponsored
More from ML engineering pipelines
- Bayesian Optimization in Python: A Simplified Mock Implementation medium
- Build a Data Helper Class in Python for ML Pipelines easy
- Build a Mock Random Forest Classifier in Python easy
- Champion Challenger Deployment Mock in Python easy
- Compare Model A vs Model B Metrics in Python easy
- Create a Minimal Great Expectations Suite Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.