Reference library

Database scaling & optimization

Indexing, connection pooling, read replicas, query tuning, and throughput-aware SQL.

5 matches

Consistent Hashing with Virtual Buckets in Python

This code maps many virtual buckets onto a few physical buckets using a consistent hashing ring, ensuring balanced distribution with minimal remapping when physical buckets change.

consistent-hashing virtual-buckets sharding

Python

import random

class VirtualBuckets:
    """Maps many virtual buckets onto few physical buckets using consistent hashing."""
    
    def __init__(self, physical_buckets, virtual_factor=100):
        self.physical = list(physical_buckets)
        self.virtual_factor = virtual_factor
        self.ring = []
        self…

13 0 Open

Database scaling & optimization easy

Geo shard by region in Python

Maps users to database shards based on geographic region with a deterministic hash fallback.

sharding geolocation database

Python

import json
from collections import defaultdict

REGION_SHARD_MAP = {
    "na": ["shard-01", "shard-02"],
    "eu": ["shard-03", "shard-04", "shard-05"],
    "ap": ["shard-06"],
    "sa": ["shard-07", "shard-08"],
}

# user_id -> region (mock lookup)
USER_REGIONS = {
    "u_1001": "na",
    "u_1002": "eu",
    "u_1003…

13 0 Open

Database scaling & optimization medium

How to Build a Shard Map Mock Dict in Python

Implement a dictionary-like class that distributes keys across multiple shards using Python's hash() for realistic data partitioning.

dict sharding hash

Python

class ShardMap:
    def __init__(self, shard_count):
        self.shards = {i: {} for i in range(shard_count)}
        self.shard_count = shard_count

    def _shard_for(self, key):
        return hash(key) % self.shard_count

    def __getitem__(self, key):
        return self.shards[self._shard_for(key)][key]

    d…

14 0 Open

Database scaling & optimization medium

How to Implement Consistent Hashing in Python

Build a consistent hash ring in Python that distributes keys across nodes and minimizes remapping when nodes are added or removed.

consistent-hashing distributed-systems sharding

Python

import hashlib
from bisect import bisect_right


class ConsistentHashRing:
    def __init__(self, nodes, replicas=3):
        self.replicas = replicas
        self.ring = {}
        self.sorted_keys = []
        for node in nodes:
            self.add_node(node)

    def _hash(self, key):
        return int(hashlib.md…

15 0 Open

Database scaling & optimization easy

How to Shard Data by User ID Hash in Python

Deterministically map user IDs to shard indexes using an MD5 hash modulo the shard count in Python.

sharding hashing md5

Python

import hashlib

def shard_id(user_id: str, num_shards: int = 4) -> int:
    """Deterministically map a user_id to a shard index using MD5."""
    digest = hashlib.md5(user_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % num_shards

if __name__ == "__main__":
    user_ids = ["alice", "bob", "carol", "d…

12 0 Open

Browse by section

Each section groups closely related Python snippets.