Hudi Upsert Mock Copy on Write in Python

Simulates Apache Hudi's Copy-on-Write upsert behavior by merging update records into a deep copy of base records, replacing matches or appending new ones.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 14 views 0 copies

Python code

27 lines
Python 3.9+
import copy
from typing import Dict, List, Any

def upsert_copy_on_write(base_records: List[Dict[str, Any]], updates: List[Dict[str, Any]], key_field: str = "id") -> List[Dict[str, Any]]:
    """Simulate Hudi Copy-on-Write upsert: merge updates into a copy of base records."""
    result = copy.deepcopy(base_records)
    update_map = {record[key_field]: record for record in updates}

    for i, record in enumerate(result):
        if record[key_field] in update_map:
            result[i] = update_map.pop(record[key_field])

    result.extend(update_map.values())
    return result

if __name__ == "__main__":
    base = [
        {"id": 1, "name": "Alice", "age": 30},
        {"id": 2, "name": "Bob", "age": 25},
        {"id": 3, "name": "Charlie", "age": 35},
    ]
    changes = [
        {"id": 2, "name": "Robert", "age": 26},
        {"id": 4, "name": "Diana", "age": 28},
    ]
    merged = upsert_copy_on_write(base, changes)
    print(merged)

Output

stdout
[{'id': 1, 'name': 'Alice', 'age': 30}, {'id': 2, 'name': 'Robert', 'age': 26}, {'id': 3, 'name': 'Charlie', 'age': 35}, {'id': 4, 'name': 'Diana', 'age': 28}]

How it works

The copy.deepcopy creates an independent copy of base records so the original list remains untouched, mimicking Hudi's immutability on write. The updates are indexed into a map keyed by the primary key field for O(1) lookups. Existing records are replaced in place using enumeration, and unmatched updates (new records) are appended at the end of the result. This mirrors Copy-on-Write semantics where the entire file is rewritten with changes applied.

Common mistakes

  • Forgetting `deepcopy` and mutating the original base list, breaking the Copy-on-Write invariant.
  • Assuming `key_field` exists in every record without validation; a missing key raises a KeyError.
  • Handling duplicate keys within updates — the map silently keeps the last occurrence.

Variations

  1. Use `merge_by_key` with dict comprehension and list comprehension for a one-liner approach.
  2. Add conflict-resolution logic (e.g., last-write-wins based on a timestamp field instead of simple replacement).

Real-world use cases

  • Mocking Hudi table upserts in unit tests before integrating with real Spark jobs on data lakes.
  • Prototyping data reconciliation logic where CDC streams need to merge updates into a snapshot.
  • Implementing a lightweight in-memory upsert for small reference datasets in microservices.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.