Database scaling & optimization
Indexing, connection pooling, read replicas, query tuning, and throughput-aware SQL.
Approximate Count with HyperLogLog in Python
A mock HyperLogLog implementation uses hash-based registers to estimate cardinality of large datasets with sublinear memory.
import hashlib
class HyperLogLog:
def __init__(self, precision=4):
if precision < 4 or precision > 16:
raise ValueError("precision must be between 4 and 16")
self.precision = precision
self.registers = [0] * (1 << precision)
def _hash(self, value):
return int(hashl…B-Tree Insert and In-Order Traversal in Python
Simulates a B-tree (order 2) with insert and split logic, then prints keys in sorted order via in-order traversal.
class BTreeNode:
def __init__(self, leaf=False):
self.leaf = leaf
self.keys = []
self.children = []
def is_full(self, t):
return len(self.keys) == 2 * t - 1
class BTree:
def __init__(self, t=2):
self.t = t
self.root = BTreeNode(leaf=True)
def insert(s…Broadcast a Small Reference Table in Python
Simulates SQL-style broadcasting of a small lookup table against a larger fact table in memory for mockups or load tests.
import random
def broadcast_mock(target, source, columns):
result = {}
for col in columns:
if col in target and col in source:
result[col] = target[col] + [source[col][i % len(source[col])] for i in range(len(target[col]))]
elif col in target:
result[col] = target[col]
…Build a Full Text Search Index in Python
Create a simple inverted index for full-text search with the standard library, supporting multi-word AND queries across documents.
import re
from collections import defaultdict
class SimpleTextIndex:
def __init__(self):
self.index = defaultdict(list)
self.documents = {}
def add_document(self, doc_id, text):
self.documents[doc_id] = text
words = set(re.findall(r'\w+', text.lower()))
for word in wo…Build a Partial Index Mock in Python for Database Filtering
Simulate a partial database index by filtering keys with a predicate, then return a limited mock lookup dictionary.
data = [
"alpha", "beta", "gamma", "delta", "epsilon",
"zeta", "eta", "theta", "iota", "kappa"
]
filtered_keys = [item for item in data if len(item) >= 5]
def mock_partial_index(keys, filter_func, limit=3):
result = {}
for key in keys:
if not filter_func(key):
continue
res…Composite index leftmost prefix in Python
Simulate a composite index in SQLite and check whether query columns match the leftmost prefix rule for index usage.
import sqlite3
def get_indexed_columns(table_name):
"""Simulate a composite index by reading column names that start with 'idx_'."""
conn = sqlite3.connect(":memory:")
conn.execute(f"CREATE TABLE {table_name} (id INTEGER, idx_col1 TEXT, idx_col2 INTEGER, other TEXT)")
conn.execute(f"CREATE INDEX idx_…Consistent Hashing with Virtual Buckets in Python
This code maps many virtual buckets onto a few physical buckets using a consistent hashing ring, ensuring balanced distribution with minimal remapping when physical buckets change.
import random
class VirtualBuckets:
"""Maps many virtual buckets onto few physical buckets using consistent hashing."""
def __init__(self, physical_buckets, virtual_factor=100):
self.physical = list(physical_buckets)
self.virtual_factor = virtual_factor
self.ring = []
self…Cross Shard Query Scatter Gather Mock in Python
Simulate a distributed database cross-shard query using a scatter-gather pattern with a mock Python implementation.
from dataclasses import dataclass
from typing import List, Dict
@dataclass
class NodeResponse:
node_id: int
data: Dict[str, float]
def mock_query_shard(shard_id: int, shard_data: Dict[str, float], query: str) -> NodeResponse:
"""Simulate querying a single shard, returning matches whose value > 50."""
…Database Helper in Python with SQLite Scaling Optimization
Build a beginner-friendly SQLite database helper class with WAL, indexed queries, and efficient batch inserts for scaling.
import sqlite3
from contextlib import contextmanager
class DatabaseHelper:
"""Beginner-friendly helper for SQLite database operations with scaling tips."""
def __init__(self, db_path):
self.db_path = db_path
@contextmanager
def connection(self):
"""Context manager for automatic comm…Database indexing and query timing optimization in Python
Create SQLite indexes and time query performance to measure speedup for large table lookups in Python.
import sqlite3
import time
def time_query(db_path, query, params=()):
conn = sqlite3.connect(db_path)
conn.execute("PRAGMA journal_mode = WAL")
start = time.perf_counter()
result = conn.execute(query, params).fetchall()
elapsed = time.perf_counter() - start
conn.close()
return result, ela…Geo shard by region in Python
Maps users to database shards based on geographic region with a deterministic hash fallback.
import json
from collections import defaultdict
REGION_SHARD_MAP = {
"na": ["shard-01", "shard-02"],
"eu": ["shard-03", "shard-04", "shard-05"],
"ap": ["shard-06"],
"sa": ["shard-07", "shard-08"],
}
# user_id -> region (mock lookup)
USER_REGIONS = {
"u_1001": "na",
"u_1002": "eu",
"u_1003…Hash index equality mock concept in Python
A simple hash index class in Python that stores key-value pairs in buckets and demonstrates basic equality-based lookup.
class HashIndex:
def __init__(self):
self._buckets = {}
def insert(self, key, value):
"""Insert a key-value pair into the hash index."""
index = hash(key) % 10
if index not in self._buckets:
self._buckets[index] = []
self._buckets[index].append((key, value))…How to Avoid SELECT * and Mock SQL Column Queries in Python
Mock a SQLite cursor to verify that queries specify explicit columns instead of using SELECT *.
import sqlite3
from unittest.mock import Mock, patch
def get_user_emails(connection):
"""Fetch only the required columns instead of SELECT *."""
cursor = connection.cursor()
cursor.execute("SELECT email FROM users")
return [row[0] for row in cursor.fetchall()]
def test_get_user_emails_specific_colu…How to Batch Load JSON Data in Python for Database Optimization
This code parses JSON data into records and loads them in batches to simulate efficient database insertion, reducing load and improving performance.
import json
import time
def parse_and_load(data, batch_size=100):
"""
Parse JSON data and batch-load into a list of dicts.
Demonstrates batching for database efficiency.
"""
records = json.loads(data)
batches = []
for i in range(0, len(records), batch_size):
batch = records[i:i + …How to Build a Connection Pool Reuse Mock in Python
Build a mock connection pool with context manager to track connection reuse, acquires, and releases in Python.
import time
from contextlib import contextmanager
class Connection:
def __init__(self, name):
self.name = name
self.in_use = False
self.busy_since = None
def fetch(self):
return f"data from {self.name}"
class ConnectionPool:
def __init__(self, size=3):
self.conn…How to Build a Shard Map Mock Dict in Python
Implement a dictionary-like class that distributes keys across multiple shards using Python's hash() for realistic data partitioning.
class ShardMap:
def __init__(self, shard_count):
self.shards = {i: {} for i in range(shard_count)}
self.shard_count = shard_count
def _shard_for(self, key):
return hash(key) % self.shard_count
def __getitem__(self, key):
return self.shards[self._shard_for(key)][key]
d…How to Convert Data with Scaling for Database Optimization in Python
A beginner-friendly helper that normalizes and scales numeric fields in a list of dicts, reducing storage footprint for database efficiency.
import json
from datetime import datetime
def convert_data(data: list[dict], scale_factor: int = 1) -> list[dict]:
"""Convert a list of dicts to a scaled, normalized format for database efficiency."""
converted = []
for row in data:
normalized = {}
for key, value in row.items():
…How to Count Star vs Estimate Matches in Python
Count how many times 'star' and 'estimate' annotations match their actual labels in a list of mock comparison results.
def count_star_vs_estimate(mock_scores):
"""
Count the number of times 'star' wins and 'estimate' wins
from a list of mock comparison results.
Args:
mock_scores: list of tuples, each (annotation, actual)
where annotation is 'star' or 'estimate'
Returns:
dict w…How to Create a Covering Index with INCLUDE Columns in Python
Create a covering index with INCLUDE columns in SQLite from Python and inspect the query plan to confirm the index covers the query.
import sqlite3
def create_covering_index_mock():
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
cursor.execute("""
CREATE TABLE employees (
id INTEGER PRIMARY KEY,
name TEXT,
department TEXT,
salary INTEGER
)
""")
employe…How to Create a Data Helper Class in Python for JSON Files
Build a beginner-friendly Python helper class to read, write, filter, and summarize JSON data files with clean, reusable methods.
import json
from pathlib import Path
class DataHelper:
"""Simple beginner-friendly helper for reading and writing JSON data files."""
@staticmethod
def read_json(filename):
file_path = Path(filename)
if file_path.exists():
with file_path.open("r", encoding="utf-8") as f:
…How to Create a Database Helper Class for Beginners in Python
Build a beginner-friendly SQLite helper class with indexing and batch inserts to optimize database queries in Python.
import sqlite3
class DatabaseHelper:
def __init__(self, db_path):
self.connection = sqlite3.connect(db_path)
self.cursor = self.connection.cursor()
def create_table_with_index(self, table_name, columns, indexed_column):
columns_sql = ", ".join(f"{name} {dtype}" for name, dtype in col…How to Eager Load with JOIN to Reduce N+1 Queries in Python
Demonstrates eager loading with a SQL JOIN to reduce N+1 query patterns down to a single database call when fetching related data.
import sqlite3
def eager_load_join_reduce(mock_db_path=":memory:"):
"""Demonstrate eager loading where joins reduce query count from N+1 to 1."""
conn = sqlite3.connect(mock_db_path)
cursor = conn.cursor()
cursor.executescript(
"""
CREATE TABLE authors (id INTEGER PRIMARY KEY, name TE…How to Explain SQLite Query Plans in Python
Build a Python function that runs EXPLAIN QUERY PLAN on SQLite in-memory tables and prints the optimizer's execution plan for any SELECT statement.
import sqlite3
def explain_query(sql: str) -> str:
"""Return the SQLite query plan for the given SQL statement."""
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
# Create sample data for a realistic plan
cursor.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
c…How to Implement Consistent Hashing in Python
Build a consistent hash ring in Python that distributes keys across nodes and minimizes remapping when nodes are added or removed.
import hashlib
from bisect import bisect_right
class ConsistentHashRing:
def __init__(self, nodes, replicas=3):
self.replicas = replicas
self.ring = {}
self.sorted_keys = []
for node in nodes:
self.add_node(node)
def _hash(self, key):
return int(hashlib.md…Browse by section
Each section groups closely related Python snippets.
Database scaling & optimization — Python code examples
What you will find here
This page collects database scaling & optimization snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.