Idempotent Writes for Sharded Databases in Python

Implement a mock shard with idempotent write support using request IDs to prevent duplicate writes and track the latest value per key.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 13 views 0 copies

idempotency sharding distributed-systems database mock

Python code

57 lines

Python 3.9+

import json


class ShardMock:
    """Mock distributed shard with idempotent write support."""

    def __init__(self, shard_id):
        self.shard_id = shard_id
        self._store = {}

    def write(self, key, value, request_id):
        """Write value only if request_id not yet processed; idempotent."""
        if request_id in self._store:
            return False, self._store[request_id]
        self._store[request_id] = value
        return True, value

    def get(self, key):
        """Retrieve the latest value for a key from processed writes."""
        # In a real system, this would scan writes and resolve conflicts.
        # For the mock, we track key→value via the write sequence.
        latest = self._store.get(f"latest_{key}")
        return latest

    def write_key(self, key, value, request_id):
        """Idempotent write that tracks latest value per key."""
        # Use key-specific storage to keep it simple.
        storage_key = f"key_{key}"
        existing = self._store.get(storage_key, {})

        if request_id in existing:
            return False, existing[request_id]

        existing[request_id] = value
        self._store[storage_key] = existing
        self._store[f"latest_{key}"] = value
        self._store[f"last_req_{key}"] = request_id
        return True, value


if __name__ == "__main__":
    shard = ShardMock(shard_id=0)

    # First write succeeds
    success, result = shard.write_key("user:42", {"name": "Alice"}, request_id="req-1")
    print(f"Write 1: success={success}, value={json.dumps(result)}")

    # Retry with same request_id → idempotent, no change
    success, result = shard.write_key("user:42", {"name": "Alice"}, request_id="req-1")
    print(f"Write 2 (retry): success={success}, value={json.dumps(result)}")

    # New request_id with different data → allowed
    success, result = shard.write_key("user:42", {"name": "Bob"}, request_id="req-2")
    print(f"Write 3: success={success}, value={json.dumps(result)}")

    # Read latest
    print(f"Latest value: {json.dumps(shard.get('user:42'))}")

Output

stdout

Write 1: success=True, value={"name": "Alice"}
Write 2 (retry): success=False, value={"name": "Alice"}
Write 3: success=True, value={"name": "Bob"}
Latest value: {"name": "Bob"}

How it works

The key insight is using a request_id as a deduplication token: the first write with a given ID stores the value and returns True, while subsequent retries with the same ID return False and the previously stored value, making writes idempotent. The mock maintains a separate storage keyed by key_{key} that maps request_id to values, plus latest_{key} and last_req_{key} to track the most recent write. This pattern mirrors how distributed systems handle at-least-once delivery by allowing consumers to safely retry writes without causing duplicates or conflicts. In production, you'd persist this state across network replicas, but the logic for deduplication remains the same.

Common mistakes

Using the key itself as the deduplication factor instead of a separate request_id, which prevents legitimate updates
Not returning the stored value on retry, breaking idempotency semantics for read-after-write patterns
Failing to persist request records before returning success, risking duplicate writes on system crashes

Variations

Use a database UNIQUE constraint on (shard_id, request_id) for stronger idempotency guarantees
Store request_id in a separate deduplication table with TTL for cleanup

Real-world use cases

Retrying failed database writes in distributed microservices without double-inserting records.
Handling at-least-once message delivery from queues like Kafka or SQS where consumers may process the same event twice.
Implementing idempotent payment processors that must not charge a customer twice for the same request.

Idempotent Writes for Sharded Databases in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Keep learning

Tutorials

Quizzes