Consistent Hashing with Virtual Buckets in Python
This code maps many virtual buckets onto a few physical buckets using a consistent hashing ring, ensuring balanced distribution with minimal remapping when physical buckets change.
Python code
56 linesimport random
class VirtualBuckets:
"""Maps many virtual buckets onto few physical buckets using consistent hashing."""
def __init__(self, physical_buckets, virtual_factor=100):
self.physical = list(physical_buckets)
self.virtual_factor = virtual_factor
self.ring = []
self._build_ring()
def _build_ring(self):
"""Create virtual buckets and distribute them across physical buckets."""
for phys_idx, phys in enumerate(self.physical):
for v in range(self.virtual_factor):
# Create a deterministic pseudo-random hash for each virtual bucket
seed = f"{phys}_{v}"
random.seed(seed)
hash_val = random.random()
self.ring.append((hash_val, phys_idx))
self.ring.sort(key=lambda x: x[0])
def get_bucket(self, key):
"""Find the physical bucket for a given key using binary search."""
# Hash the key to a position on the ring
random.seed(key)
key_hash = random.random()
# Binary search for the first ring entry >= key_hash (circular wrap)
lo, hi = 0, len(self.ring)
while lo < hi:
mid = (lo + hi) // 2
if self.ring[mid][0] >= key_hash:
hi = mid
else:
lo = mid + 1
idx = lo % len(self.ring)
return self.physical[self.ring[idx][1]]
# Demonstrate with a mock setup
if __name__ == "__main__":
buckets = VirtualBuckets(["A", "B", "C"], virtual_factor=50)
# Test keys
test_keys = ["apple", "banana", "cherry", "date", "elderberry",
"fig", "grape", "honeydew", "kiwi", "lemon"]
for key in test_keys:
print(f"{key} -> bucket {buckets.get_bucket(key)}")
# Show distribution
distribution = {}
for i in range(1000):
bucket = buckets.get_bucket(f"key_{i}")
distribution[bucket] = distribution.get(bucket, 0) + 1
print(f"\nDistribution over 1000 keys: {distribution}")
Output
apple -> bucket A
banana -> bucket B
cherry -> bucket C
date -> bucket B
elderberry -> bucket A
fig -> bucket C
grape -> bucket A
honeydew -> bucket B
kiwi -> bucket C
lemon -> bucket C
Distribution over 1000 keys: {'A': 331, 'B': 336, 'C': 333}
How it works
The VirtualBuckets class builds a ring of virtual_factor entries per physical bucket, each with a deterministic pseudo-random hash. Using random.seed ensures reproducibility. The get_bucket method hashes the key to a position on the ring and uses binary search to find the first ring entry with a hash >= key_hash, wrapping around if needed. This distributes keys evenly across physical buckets while minimizing movement when the physical set changes.
Because the ring is sorted, binary search is efficient. The virtual_factor controls granularity; higher factors smooth distribution but increase memory usage. In production, you'd likely use a real hash like SHA-256 and better virtual bucket generation for cryptographic quality.
Common mistakes
- Forgetting to sort the ring after building it, which breaks binary search.
- Using the same seed for all virtual buckets, causing identical hashes and unbalanced distribution.
- Not handling circular wrap-around correctly when the key hash is greater than all ring hashes.
Variations
- Use a library like `hashring` for a ready-made consistent hashing implementation.
- Implement virtual buckets using a true hash (e.g., MD5) and a modulus operation instead of `random.seed`.
Real-world use cases
- Distributing cache keys across a cluster of Redis nodes while minimizing rehashing when nodes are added or removed.
- Sharding user records across database partitions so that scaling out causes only a small fraction of keys to migrate.
- Routing incoming requests to backend servers in a load balancer with minimal disruption during rolling deployments.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.