Database Helper in Python with SQLite Scaling Optimization

Build a beginner-friendly SQLite database helper class with WAL, indexed queries, and efficient batch inserts for scaling.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 14 views 0 copies

Python code

66 lines
Python 3.9+
import sqlite3
from contextlib import contextmanager


class DatabaseHelper:
    """Beginner-friendly helper for SQLite database operations with scaling tips."""

    def __init__(self, db_path):
        self.db_path = db_path

    @contextmanager
    def connection(self):
        """Context manager for automatic commit/rollback and closing."""
        conn = sqlite3.connect(self.db_path)
        conn.execute("PRAGMA journal_mode=WAL")  # Better concurrency
        conn.execute("PRAGMA synchronous=NORMAL")  # Faster writes
        try:
            yield conn
            conn.commit()
        except Exception:
            conn.rollback()
            raise
        finally:
            conn.close()

    def create_sample_table(self):
        """Create a simple indexed table for demonstration."""
        with self.connection() as conn:
            conn.execute("""
                CREATE TABLE IF NOT EXISTS users (
                    id INTEGER PRIMARY KEY,
                    name TEXT NOT NULL,
                    email TEXT UNIQUE NOT NULL,
                    age INTEGER
                )
            """)
            conn.execute("CREATE INDEX IF NOT EXISTS idx_users_age ON users(age)")

    def insert_many(self, users):
        """Insert multiple rows efficiently using executemany."""
        with self.connection() as conn:
            conn.executemany(
                "INSERT INTO users (name, email, age) VALUES (?, ?, ?)",
                users
            )

    def query_by_age(self, min_age):
        """Query using the index — faster on large datasets."""
        with self.connection() as conn:
            return list(conn.execute(
                "SELECT name, email FROM users WHERE age >= ?",
                (min_age,)
            ))


if __name__ == "__main__":
    db = DatabaseHelper(":memory:")  # in-memory for demo
    db.create_sample_table()
    db.insert_many([
        ("Alice", "alice@example.com", 30),
        ("Bob", "bob@example.com", 25),
        ("Carol", "carol@example.com", 40)
    ])
    results = db.query_by_age(28)
    for row in results:
        print(row)

Output

stdout
('Alice', 'alice@example.com')
('Carol', 'carol@example.com')

How it works

The contextmanager decorator wraps sqlite3.connect so every transaction commits on success and rolls back on errors automatically. PRAGMA journal_mode=WAL allows concurrent reads while writes happen, and synchronous=NORMAL reduces disk sync overhead for faster performance on busy workloads. The executemany method batches inserts into a single operation, which is far quicker than row-by-row inserts. Creating an index on age means queries like WHERE age >= ? can use a B-tree lookup instead of a full table scan, which scales better as the table grows.

Common mistakes

  • Forgetting to call `conn.close()` manually, which the context manager handles automatically
  • Using `executemany` with a list of tuples but mismatching the number of placeholders
  • Creating indexes after inserting large amounts of data instead of before
  • Enabling WAL mode on every new connection when it should be set once on the database file

Variations

  1. Switch to `sqlite3.connect(self.db_path, isolation_level=None)` for autocommit mode
  2. Use `conn.row_factory = sqlite3.Row` to access columns by name instead of by index

Real-world use cases

  • A small CRM tool that stores customer records and needs fast age-based filtering for marketing segments.
  • An analytics dashboard caching processed metrics in a local SQLite file with concurrent report readers.
  • A batch ETL job inserting thousands of user rows nightly into an indexed SQLite staging table.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.