Cross Shard Query Scatter Gather Mock in Python
Simulate a distributed database cross-shard query using a scatter-gather pattern with a mock Python implementation.
Python code
42 linesfrom dataclasses import dataclass
from typing import List, Dict
@dataclass
class NodeResponse:
node_id: int
data: Dict[str, float]
def mock_query_shard(shard_id: int, shard_data: Dict[str, float], query: str) -> NodeResponse:
"""Simulate querying a single shard, returning matches whose value > 50."""
matches = {k: v for k, v in shard_data.items() if query in k and v > 50}
return NodeResponse(node_id=shard_id, data=matches)
def scatter_query(shards: List[Dict[str, float]], query: str) -> List[NodeResponse]:
"""Send query to all shards (scatter phase)."""
return [mock_query_shard(i, shard, query) for i, shard in enumerate(shards)]
def gather_results(responses: List[NodeResponse]) -> Dict[str, float]:
"""Merge all node responses (gather phase) into a single result dict."""
merged: Dict[str, float] = {}
for resp in responses:
merged.update(resp.data)
return merged
def scatter_gather_mock(shards: List[Dict[str, float]], query: str) -> Dict[str, float]:
"""Full cross-shard query: scatter → gather."""
return gather_results(scatter_query(shards, query))
if __name__ == "__main__":
shards = [
{"alpha": 20, "alpine": 90, "beta": 70},
{"alpha2": 80, "beta2": 30, "alpha3": 100},
{"alpine2": 60, "beta3": 95},
]
result = scatter_gather_mock(shards, "alpha")
print(result)
Output
{'alpine': 90, 'alpha2': 80, 'alpha3': 100}
How it works
The scatter_query function simulates sending a query to every shard in parallel, mocking the distributed nature of a database cluster. Each shard returns a NodeResponse dataclass containing partial matches based on the criteria (key contains 'alpha' and value > 50). The gather_results function then merges all partial results into a unified dictionary, preserving the highest-level view of the data. This pattern mimics MapReduce-style fan-out/fan-in where computation is distributed then aggregated. Using dataclasses keeps the node metadata clean and type-safe, while comprehensions keep the implementation concise.
Common mistakes
- Assuming shards are queried truly in parallel when using sequential list comprehension
- Handling duplicate keys across shards inconsistently (last write wins in this implementation)
- Ignoring per-shard failure scenarios that would need retry logic in a real system
- Filtering on the wrong side (e.g., filtering after merge instead of at shard level)
Variations
- Use `concurrent.futures.ThreadPoolExecutor` to actually parallelize scatter phase
- Return partial errors with per-shard status for more realistic distributed behavior
Real-world use cases
- Querying a Redis cluster or Elasticsearch where results from multiple shards need merging
- Implementing a search across partitioned database tables for an e-commerce product catalog
- Building a distributed analytics system that aggregates clickstream data from multiple nodes
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.