Reference library

Database scaling & optimization

Indexing, connection pooling, read replicas, query tuning, and throughput-aware SQL.

5 matches

Broadcast a Small Reference Table in Python

Simulates SQL-style broadcasting of a small lookup table against a larger fact table in memory for mockups or load tests.

broadcast mock-data data-engineering

Python

import random

def broadcast_mock(target, source, columns):
    result = {}
    for col in columns:
        if col in target and col in source:
            result[col] = target[col] + [source[col][i % len(source[col])] for i in range(len(target[col]))]
        elif col in target:
            result[col] = target[col]
…

15 0 Open

Database scaling & optimization easy

How to Batch Load JSON Data in Python for Database Optimization

This code parses JSON data into records and loads them in batches to simulate efficient database insertion, reducing load and improving performance.

json batching database

Python

import json
import time

def parse_and_load(data, batch_size=100):
    """
    Parse JSON data and batch-load into a list of dicts.
    Demonstrates batching for database efficiency.
    """
    records = json.loads(data)
    batches = []

    for i in range(0, len(records), batch_size):
        batch = records[i:i + …

12 0 Open

Database scaling & optimization medium

How to Eager Load with JOIN to Reduce N+1 Queries in Python

Demonstrates eager loading with a SQL JOIN to reduce N+1 query patterns down to a single database call when fetching related data.

eager-loading n-plus-1 join

Python

import sqlite3


def eager_load_join_reduce(mock_db_path=":memory:"):
    """Demonstrate eager loading where joins reduce query count from N+1 to 1."""
    conn = sqlite3.connect(mock_db_path)
    cursor = conn.cursor()
    cursor.executescript(
        """
        CREATE TABLE authors (id INTEGER PRIMARY KEY, name TE…

16 0 Open

Database scaling & optimization medium

How to Implement Consistent Hashing in Python

Build a consistent hash ring in Python that distributes keys across nodes and minimizes remapping when nodes are added or removed.

consistent-hashing distributed-systems sharding

Python

import hashlib
from bisect import bisect_right


class ConsistentHashRing:
    def __init__(self, nodes, replicas=3):
        self.replicas = replicas
        self.ring = {}
        self.sorted_keys = []
        for node in nodes:
            self.add_node(node)

    def _hash(self, key):
        return int(hashlib.md…

15 0 Open

Database scaling & optimization easy

Route SELECT Queries to Read Replicas in Python

A mock round-robin router that forwards SELECT queries to read replicas and sends writes to the primary.

database replication routing

Python

import random

class ReadReplicaRouter:
    """Round-robin router that sends SELECT queries to read replicas."""
    
    def __init__(self, replicas):
        self.replicas = replicas
        self.counter = 0
    
    def route(self, sql):
        if sql.strip().upper().startswith("SELECT"):
            replica = sel…

13 0 Open

Browse by section

Each section groups closely related Python snippets.