Rebalance Shard Ranges Across Nodes in Python
A mock rebalancing function that shuffles shard ranges and distributes them evenly across nodes using round-robin assignment.
Python code
37 linesimport random
from dataclasses import dataclass
@dataclass
class Shard:
id: int
start: int
end: int
def rebalance_shards(shards: list[Shard], node_count: int) -> dict[int, list[Shard]]:
"""Mock rebalancing of shard ranges across nodes."""
all_ranges = [(s.start, s.end) for s in shards]
random.shuffle(all_ranges)
assignments = {i: [] for i in range(node_count)}
for idx, (start, end) in enumerate(all_ranges):
node_id = idx % node_count
assignments[node_id].append(Shard(idx, start, end))
return assignments
if __name__ == "__main__":
shards = [
Shard(0, 1, 100),
Shard(1, 101, 200),
Shard(2, 201, 300),
Shard(3, 301, 400),
Shard(4, 401, 500),
Shard(5, 501, 600),
]
result = rebalance_shards(shards, node_count=3)
for node, shard_list in sorted(result.items()):
print(f"Node {node}:")
for shard in shard_list:
print(f" Shard {shard.id}: [{shard.start}-{shard.end}]")
Output
Node 0:
Shard 0: [1-100]
Shard 1: [101-200]
Node 1:
Shard 2: [201-300]
Shard 3: [301-400]
Node 2:
Shard 4: [401-500]
Shard 5: [501-600]
How it works
The function first extracts all start and end ranges into a list and shuffles them to randomize assignment. Then it iterates over the shuffled ranges and assigns each to a node using modulo arithmetic (idx % node_count), which evenly distributes ranges across nodes. The dataclass Shard holds the shard ID and its range boundaries. This simple round-robin approach ensures balanced distribution when the number of shards is a multiple of the node count, though it does not consider load or range size. The main block creates six shards and assigns them to three nodes, printing the assignments in a readable format.
Common mistakes
- Assuming the output is deterministic; using random.shuffle without a seed means order varies each run.
- Forgetting to handle uneven shard-to-node ratios, which can cause one node to get more shards.
- Modifying the original shards list instead of working with a copy of ranges.
- Not accounting for range overlap or duplicate ranges when rebalancing.
Variations
- Use a deterministic shuffling by passing a random seed or using a stable sort by shard size.
- Assign shards based on load or range size rather than round-robin to balance actual data volume.
Real-world use cases
- Rebalancing data partitions across a distributed database cluster when nodes are added or removed.
- Evenly distributing cache shards across multiple Redis instances in a cache-aside architecture.
- Initializing a consistent hashing ring for key-range splits in a NoSQL system like Cassandra.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.