Hudi Upsert Mock Copy on Write in Python
Simulates Apache Hudi's Copy-on-Write upsert behavior by merging update records into a deep copy of base records, replacing matches or appending new ones.
Python code
27 linesimport copy
from typing import Dict, List, Any
def upsert_copy_on_write(base_records: List[Dict[str, Any]], updates: List[Dict[str, Any]], key_field: str = "id") -> List[Dict[str, Any]]:
"""Simulate Hudi Copy-on-Write upsert: merge updates into a copy of base records."""
result = copy.deepcopy(base_records)
update_map = {record[key_field]: record for record in updates}
for i, record in enumerate(result):
if record[key_field] in update_map:
result[i] = update_map.pop(record[key_field])
result.extend(update_map.values())
return result
if __name__ == "__main__":
base = [
{"id": 1, "name": "Alice", "age": 30},
{"id": 2, "name": "Bob", "age": 25},
{"id": 3, "name": "Charlie", "age": 35},
]
changes = [
{"id": 2, "name": "Robert", "age": 26},
{"id": 4, "name": "Diana", "age": 28},
]
merged = upsert_copy_on_write(base, changes)
print(merged)
Output
[{'id': 1, 'name': 'Alice', 'age': 30}, {'id': 2, 'name': 'Robert', 'age': 26}, {'id': 3, 'name': 'Charlie', 'age': 35}, {'id': 4, 'name': 'Diana', 'age': 28}]
How it works
The copy.deepcopy creates an independent copy of base records so the original list remains untouched, mimicking Hudi's immutability on write. The updates are indexed into a map keyed by the primary key field for O(1) lookups. Existing records are replaced in place using enumeration, and unmatched updates (new records) are appended at the end of the result. This mirrors Copy-on-Write semantics where the entire file is rewritten with changes applied.
Common mistakes
- Forgetting `deepcopy` and mutating the original base list, breaking the Copy-on-Write invariant.
- Assuming `key_field` exists in every record without validation; a missing key raises a KeyError.
- Handling duplicate keys within updates — the map silently keeps the last occurrence.
Variations
- Use `merge_by_key` with dict comprehension and list comprehension for a one-liner approach.
- Add conflict-resolution logic (e.g., last-write-wins based on a timestamp field instead of simple replacement).
Real-world use cases
- Mocking Hudi table upserts in unit tests before integrating with real Spark jobs on data lakes.
- Prototyping data reconciliation logic where CDC streams need to merge updates into a snapshot.
- Implementing a lightweight in-memory upsert for small reference datasets in microservices.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.