Simulate Shard Key Cardinality in Python

Generate mock data with configurable cardinality to evaluate shard key distribution and detect hotspots in database scaling design.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 17 views 0 copies

Python code

35 lines
Python 3.9+
import random
import string

def calculate_cardinality(values):
    """Return the number of distinct values in the given list."""
    return len(set(values))

def generate_mock_data(num_records, cardinality):
    """Generate mock records for a shard key with given cardinality."""
    possible_keys = [f"key_{i:04d}" for i in range(cardinality)]
    return [random.choice(possible_keys) for _ in range(num_records)]

def evaluate_shard_key(cardinality, num_records=10000):
    """Simulate distribution of a shard key across records."""
    records = generate_mock_data(num_records, cardinality)
    distinct = calculate_cardinality(records)
    distribution = {}
    for key in records:
        distribution[key] = distribution.get(key, 0) + 1

    avg_per_shard = num_records / distinct
    return {
        "requested_cardinality": cardinality,
        "actual_distinct": distinct,
        "total_records": num_records,
        "avg_records_per_key": round(avg_per_shard, 2),
        "max_records_for_single_key": max(distribution.values()),
        "min_records_for_single_key": min(distribution.values()),
    }

if __name__ == "__main__":
    random.seed(42)  # deterministic output
    for card in [10, 100, 1000]:
        result = evaluate_shard_key(card)
        print(f"Cardinality={card}: {result}")

Output

stdout
Cardinality=10: {'requested_cardinality': 10, 'actual_distinct': 10, 'total_records': 10000, 'avg_records_per_key': 1000.0, 'max_records_for_single_key': 1061, 'min_records_for_single_key': 930}
Cardinality=100: {'requested_cardinality': 100, 'actual_distinct': 100, 'total_records': 10000, 'avg_records_per_key': 100.0, 'max_records_for_single_key': 116, 'min_records_for_single_key': 75}
Cardinality=1000: {'requested_cardinality': 1000, 'actual_distinct': 1000, 'total_records': 10000, 'avg_records_per_key': 10.0, 'max_records_for_single_key': 22, 'min_records_for_single_key': 2}

How it works

This script simulates shard key behavior by generating random records with a configurable cardinality and analyzing their distribution. The random.seed(42) call ensures deterministic output, making results reproducible across runs. The distribution metrics — max and min records per key — reveal potential hotspots: low cardinality creates uneven loads, while high cardinality spreads data evenly. In real sharding, high cardinality with low skew reduces the risk that a single shard becomes a bottleneck, matching the insight shown here.

Common mistakes

  • Using low cardinality values that create uneven distribution and hotspots
  • Not seeding random for reproducible benchmark results
  • Confusing cardinality with the actual number of shards in the cluster
  • Ignoring skew — max records per key shows the real load imbalance

Variations

  1. Use `Counter` from collections for cleaner distribution counting
  2. Add distribution skew by weighting keys with a Zipfian distribution for realism

Real-world use cases

  • Validating shard key choices before provisioning database clusters in production
  • Modeling read-heavy workloads to predict which shards will receive uneven load
  • Capacity planning — estimating per-shard storage and query latency for new tables

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.