Cross Shard Query Scatter Gather Mock in Python

Simulate a distributed database cross-shard query using a scatter-gather pattern with a mock Python implementation.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 13 views 0 copies

Python code

42 lines
Python 3.9+
from dataclasses import dataclass
from typing import List, Dict


@dataclass
class NodeResponse:
    node_id: int
    data: Dict[str, float]


def mock_query_shard(shard_id: int, shard_data: Dict[str, float], query: str) -> NodeResponse:
    """Simulate querying a single shard, returning matches whose value > 50."""
    matches = {k: v for k, v in shard_data.items() if query in k and v > 50}
    return NodeResponse(node_id=shard_id, data=matches)


def scatter_query(shards: List[Dict[str, float]], query: str) -> List[NodeResponse]:
    """Send query to all shards (scatter phase)."""
    return [mock_query_shard(i, shard, query) for i, shard in enumerate(shards)]


def gather_results(responses: List[NodeResponse]) -> Dict[str, float]:
    """Merge all node responses (gather phase) into a single result dict."""
    merged: Dict[str, float] = {}
    for resp in responses:
        merged.update(resp.data)
    return merged


def scatter_gather_mock(shards: List[Dict[str, float]], query: str) -> Dict[str, float]:
    """Full cross-shard query: scatter → gather."""
    return gather_results(scatter_query(shards, query))


if __name__ == "__main__":
    shards = [
        {"alpha": 20, "alpine": 90, "beta": 70},
        {"alpha2": 80, "beta2": 30, "alpha3": 100},
        {"alpine2": 60, "beta3": 95},
    ]
    result = scatter_gather_mock(shards, "alpha")
    print(result)

Output

stdout
{'alpine': 90, 'alpha2': 80, 'alpha3': 100}

How it works

The scatter_query function simulates sending a query to every shard in parallel, mocking the distributed nature of a database cluster. Each shard returns a NodeResponse dataclass containing partial matches based on the criteria (key contains 'alpha' and value > 50). The gather_results function then merges all partial results into a unified dictionary, preserving the highest-level view of the data. This pattern mimics MapReduce-style fan-out/fan-in where computation is distributed then aggregated. Using dataclasses keeps the node metadata clean and type-safe, while comprehensions keep the implementation concise.

Common mistakes

  • Assuming shards are queried truly in parallel when using sequential list comprehension
  • Handling duplicate keys across shards inconsistently (last write wins in this implementation)
  • Ignoring per-shard failure scenarios that would need retry logic in a real system
  • Filtering on the wrong side (e.g., filtering after merge instead of at shard level)

Variations

  1. Use `concurrent.futures.ThreadPoolExecutor` to actually parallelize scatter phase
  2. Return partial errors with per-shard status for more realistic distributed behavior

Real-world use cases

  • Querying a Redis cluster or Elasticsearch where results from multiple shards need merging
  • Implementing a search across partitioned database tables for an e-commerce product catalog
  • Building a distributed analytics system that aggregates clickstream data from multiple nodes

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.