Backward Compatible Schema Evolution in Python

A mock schema validator that evolves JSON schemas while preserving backward compatibility by keeping old fields and validating required ones.

Medium Python 3.9+ Aug 9, 2026 Microservices patterns 15 views 0 copies

Python code

38 lines
Python 3.9+
import json
from copy import deepcopy


class SchemaValidator:
    def __init__(self, schema):
        self.schema = schema

    def evolve(self, new_schema):
        """Evolve mock schema while keeping backward compatibility."""
        for field in self.schema:
            if field not in new_schema:
                # Preserve old fields in evolved schema
                new_schema[field] = deepcopy(self.schema[field])
        self.schema = new_schema
        return self.schema

    def validate(self, data):
        """Mock validation: check that all required old fields exist."""
        errors = []
        for field, rules in self.schema.items():
            if rules.get("required", False) and field not in data:
                errors.append(f"Missing required field: {field}")
        return errors if errors else "Valid"


if __name__ == "__main__":
    initial_schema = {"name": {"required": True}, "age": {"required": False}}
    validator = SchemaValidator(initial_schema)

    # Simulate schema evolution adding a new field
    evolved = validator.evolve({"name": {"required": True}, "email": {"required": True}})
    print("Evolved schema:", json.dumps(evolved, indent=2))

    # Old data still validates as compatible (ignores unknown 'age')
    print("Old data:", validator.validate({"name": "Alice"}))
    print("New data:", validator.validate({"name": "Alice", "email": "alice@example.com", "age": 30}))
    print("Missing name:", validator.validate({"email": "bob@example.com"}))

Output

stdout
Evolved schema: {
  "name": {
    "required": true
  },
  "email": {
    "required": true
  },
  "age": {
    "required": false
  }
}
Old data: Valid
New data: Valid
Missing name: ['Missing required field: name']

How it works

This class keeps a schema as a dict and 'evolve' merges new fields while deep-copying old ones that aren't present, ensuring old consumers still see their fields. Validation only checks required fields exist in the payload, so extra fields like 'age' are ignored, which mimics forward-compatible parsing. Deepcopy prevents shared references between schemas, and the validator is intentionally simple to demonstrate the evolution pattern.

Common mistakes

  • Using shallow copy instead of deepcopy, leading to shared nested dicts
  • Assuming validation rejects unknown fields when it only checks required ones
  • Missing fields with required=False still need to be preserved for backward compatibility
  • Forgetting to update the internal schema reference after evolve

Variations

  1. Use a `dataclass` for schema rules to add type checking at runtime
  2. Use typing.TypedDict to describe schema structure for static analysis

Real-world use cases

  • Migrating API request payloads when adding new optional fields without breaking old clients
  • Managing config file schemas in microservices to allow rolling updates across services
  • Evolving event payloads in a Kafka streaming pipeline to keep producers and consumers compatible

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Microservices patterns

Related tutorials and quizzes for this topic.