Backward Compatible Schema Evolution in Python
A mock schema validator that evolves JSON schemas while preserving backward compatibility by keeping old fields and validating required ones.
Python code
38 linesimport json
from copy import deepcopy
class SchemaValidator:
def __init__(self, schema):
self.schema = schema
def evolve(self, new_schema):
"""Evolve mock schema while keeping backward compatibility."""
for field in self.schema:
if field not in new_schema:
# Preserve old fields in evolved schema
new_schema[field] = deepcopy(self.schema[field])
self.schema = new_schema
return self.schema
def validate(self, data):
"""Mock validation: check that all required old fields exist."""
errors = []
for field, rules in self.schema.items():
if rules.get("required", False) and field not in data:
errors.append(f"Missing required field: {field}")
return errors if errors else "Valid"
if __name__ == "__main__":
initial_schema = {"name": {"required": True}, "age": {"required": False}}
validator = SchemaValidator(initial_schema)
# Simulate schema evolution adding a new field
evolved = validator.evolve({"name": {"required": True}, "email": {"required": True}})
print("Evolved schema:", json.dumps(evolved, indent=2))
# Old data still validates as compatible (ignores unknown 'age')
print("Old data:", validator.validate({"name": "Alice"}))
print("New data:", validator.validate({"name": "Alice", "email": "alice@example.com", "age": 30}))
print("Missing name:", validator.validate({"email": "bob@example.com"}))
Output
Evolved schema: {
"name": {
"required": true
},
"email": {
"required": true
},
"age": {
"required": false
}
}
Old data: Valid
New data: Valid
Missing name: ['Missing required field: name']
How it works
This class keeps a schema as a dict and 'evolve' merges new fields while deep-copying old ones that aren't present, ensuring old consumers still see their fields. Validation only checks required fields exist in the payload, so extra fields like 'age' are ignored, which mimics forward-compatible parsing. Deepcopy prevents shared references between schemas, and the validator is intentionally simple to demonstrate the evolution pattern.
Common mistakes
- Using shallow copy instead of deepcopy, leading to shared nested dicts
- Assuming validation rejects unknown fields when it only checks required ones
- Missing fields with required=False still need to be preserved for backward compatibility
- Forgetting to update the internal schema reference after evolve
Variations
- Use a `dataclass` for schema rules to add type checking at runtime
- Use typing.TypedDict to describe schema structure for static analysis
Real-world use cases
- Migrating API request payloads when adding new optional fields without breaking old clients
- Managing config file schemas in microservices to allow rolling updates across services
- Evolving event payloads in a Kafka streaming pipeline to keep producers and consumers compatible
Sponsored
More from Microservices patterns
- BFF aggregation pattern: combine multiple service responses in Python easy
- Bulkhead Thread Pool per Service Mock in Python medium
- CQRS with Separate Read and Write Repositories in Python medium
- Cache-Aside Pattern in Python: Per-Service Mock easy
- Consumer Driven Contract Pact Mock in Python medium
- Correlation ID HTTP header mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.