Simulate Shard Key Cardinality in Python
Generate mock data with configurable cardinality to evaluate shard key distribution and detect hotspots in database scaling design.
Python code
35 linesimport random
import string
def calculate_cardinality(values):
"""Return the number of distinct values in the given list."""
return len(set(values))
def generate_mock_data(num_records, cardinality):
"""Generate mock records for a shard key with given cardinality."""
possible_keys = [f"key_{i:04d}" for i in range(cardinality)]
return [random.choice(possible_keys) for _ in range(num_records)]
def evaluate_shard_key(cardinality, num_records=10000):
"""Simulate distribution of a shard key across records."""
records = generate_mock_data(num_records, cardinality)
distinct = calculate_cardinality(records)
distribution = {}
for key in records:
distribution[key] = distribution.get(key, 0) + 1
avg_per_shard = num_records / distinct
return {
"requested_cardinality": cardinality,
"actual_distinct": distinct,
"total_records": num_records,
"avg_records_per_key": round(avg_per_shard, 2),
"max_records_for_single_key": max(distribution.values()),
"min_records_for_single_key": min(distribution.values()),
}
if __name__ == "__main__":
random.seed(42) # deterministic output
for card in [10, 100, 1000]:
result = evaluate_shard_key(card)
print(f"Cardinality={card}: {result}")
Output
Cardinality=10: {'requested_cardinality': 10, 'actual_distinct': 10, 'total_records': 10000, 'avg_records_per_key': 1000.0, 'max_records_for_single_key': 1061, 'min_records_for_single_key': 930}
Cardinality=100: {'requested_cardinality': 100, 'actual_distinct': 100, 'total_records': 10000, 'avg_records_per_key': 100.0, 'max_records_for_single_key': 116, 'min_records_for_single_key': 75}
Cardinality=1000: {'requested_cardinality': 1000, 'actual_distinct': 1000, 'total_records': 10000, 'avg_records_per_key': 10.0, 'max_records_for_single_key': 22, 'min_records_for_single_key': 2}
How it works
This script simulates shard key behavior by generating random records with a configurable cardinality and analyzing their distribution. The random.seed(42) call ensures deterministic output, making results reproducible across runs. The distribution metrics — max and min records per key — reveal potential hotspots: low cardinality creates uneven loads, while high cardinality spreads data evenly. In real sharding, high cardinality with low skew reduces the risk that a single shard becomes a bottleneck, matching the insight shown here.
Common mistakes
- Using low cardinality values that create uneven distribution and hotspots
- Not seeding random for reproducible benchmark results
- Confusing cardinality with the actual number of shards in the cluster
- Ignoring skew — max records per key shows the real load imbalance
Variations
- Use `Counter` from collections for cleaner distribution counting
- Add distribution skew by weighting keys with a Zipfian distribution for realism
Real-world use cases
- Validating shard key choices before provisioning database clusters in production
- Modeling read-heavy workloads to predict which shards will receive uneven load
- Capacity planning — estimating per-shard storage and query latency for new tables
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.