How to Simulate Colocated Shard Joins in Python

Groups shards by their node and merges co-located shards into a single logical unit, checking capacity constraints.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 11 views 0 copies

sharding database distributed-systems capacity-planning simulation

Python code

50 lines

Python 3.9+

import random
from collections import defaultdict


def simulate_colocated_shards_join(nodes: list[dict], shards: list[dict]) -> dict:
    """
    Simulates the join of co-located shards (on the same node) into a single
    logical shard. Returns the resulting node-to-shard mapping.

    Each node: {'id': str, 'capacity': int}
    Each shard: {'id': str, 'node': str, 'size': int}
    """
    node_capacity = {n['id']: n['capacity'] for n in nodes}
    shards_by_node = defaultdict(list)

    for shard in shards:
        shards_by_node[shard['node']].append(shard)

    result = {}
    for node_id, shard_list in shards_by_node.items():
        total_size = sum(s['size'] for s in shard_list)
        # Mock join: co-located shards merge into one logical unit
        result[node_id] = {
            'merged_shard_id': f"{node_id}_merged",
            'total_size': total_size,
            'within_capacity': total_size <= node_capacity[node_id],
            'co_located_count': len(shard_list),
        }

    return result


if __name__ == "__main__":
    random.seed(42)
    nodes = [
        {'id': 'node_a', 'capacity': 100},
        {'id': 'node_b', 'capacity': 150},
        {'id': 'node_c', 'capacity': 50},
    ]
    shards = [
        {'id': 's1', 'node': 'node_a', 'size': 40},
        {'id': 's2', 'node': 'node_a', 'size': 30},
        {'id': 's3', 'node': 'node_b', 'size': 10},
        {'id': 's4', 'node': 'node_b', 'size': 20},
        {'id': 's5', 'node': 'node_c', 'size': 60},
    ]

    mock_result = simulate_colocated_shards_join(nodes, shards)
    for node_id, info in sorted(mock_result.items()):
        print(f"{node_id}: {info}")

Output

stdout

node_a: {'merged_shard_id': 'node_a_merged', 'total_size': 70, 'within_capacity': True, 'co_located_count': 2}
node_b: {'merged_shard_id': 'node_b_merged', 'total_size': 30, 'within_capacity': True, 'co_located_count': 2}
node_c: {'merged_shard_id': 'node_c_merged', 'total_size': 60, 'within_capacity': False, 'co_located_count': 1}

How it works

The defaultdict(list) groups shards by their node ID, letting us handle co-located shards efficiently without checking dictionary keys manually. Summing shard sizes gives the total logical size after merging, and comparing against node capacity determines whether the merge fits. The merged_shard_id uses a deterministic naming pattern so the mock is predictable in tests. This pattern mirrors how real systems evaluate whether consolidating data onto fewer nodes stays within resource limits.

Common mistakes

Forgetting to handle nodes with zero shards in the input
Assuming shard sizes are always integers when summing
Not validating that referenced nodes exist in the nodes list

Variations

Use a plain dict with setdefault instead of defaultdict(list)
Add a random delay to simulate network latency in the join operation

Real-world use cases

Estimating storage savings when consolidating shards during cluster rebalancing.
Testing capacity planning before moving co-located shards across a production database.
Building a mock for load simulations in distributed database benchmarks.

How to Simulate Colocated Shard Joins in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Database scaling & optimization

Keep learning

Tutorials

Quizzes