Database indexing and query timing optimization in Python
Create SQLite indexes and time query performance to measure speedup for large table lookups in Python.
Python code
46 linesimport sqlite3
import time
def time_query(db_path, query, params=()):
conn = sqlite3.connect(db_path)
conn.execute("PRAGMA journal_mode = WAL")
start = time.perf_counter()
result = conn.execute(query, params).fetchall()
elapsed = time.perf_counter() - start
conn.close()
return result, elapsed
def add_index(db_path, table, column):
conn = sqlite3.connect(db_path)
index_name = f"idx_{table}_{column}"
conn.execute(f"CREATE INDEX IF NOT EXISTS {index_name} ON {table}({column})")
conn.commit()
conn.close()
return index_name
if __name__ == "__main__":
db = "scaling_optimization.db"
conn = sqlite3.connect(db)
conn.execute("DROP TABLE IF EXISTS users")
conn.execute("CREATE TABLE users (id INTEGER, email TEXT, age INTEGER)")
conn.executemany(
"INSERT INTO users VALUES (?, ?, ?)",
[(i, f"user{i}@example.com", 18 + (i % 50)) for i in range(10_000)],
)
conn.commit()
conn.close()
slow_query = "SELECT * FROM users WHERE email = ?"
slow_result, slow_time = time_query(db, slow_query, ("user5000@example.com",))
index_name = add_index(db, "users", "email")
fast_result, fast_time = time_query(db, slow_query, ("user5000@example.com",))
print(f"Slow query: {slow_time:.6f}s -> {slow_result}")
print(f"Index created: {index_name}")
print(f"Fast query: {fast_time:.6f}s -> {fast_result}")
print(f"Speedup: {slow_time / fast_time:.2f}x")
Output
Slow query: 0.002319s -> [(5000, 'user5000@example.com', 18)]
Index created: idx_users_email
Fast query: 0.000142s -> [(5000, 'user5000@example.com', 18)]
Speedup: 16.33x
How it works
The time_query function sets WAL mode for better concurrency and measures query duration with time.perf_counter, which is precise on all platforms. The add_index helper creates a covering index using CREATE INDEX IF NOT EXISTS, avoiding errors on repeated runs. With 10,000 rows where the email is unique, the index turns a full table scan into a B-tree lookup, drastically reducing time. The script demonstrates that indexing a frequently queried column yields significant performance improvements, ideal for beginner-facing database optimization lessons.
Common mistakes
- Forgetting `conn.commit()` after DDL statements like CREATE INDEX
- Not closing connections, which can cause lock contention
- Assuming indexes always help — they add write overhead
Variations
- Use `EXPLAIN QUERY PLAN` to verify the database uses the index
- Wrap timing logic in a context manager for cleaner code
- Apply composite indexes for multi-column WHERE clauses
Real-world use cases
- Benchmarking query performance after adding indexes in application databases like PostgreSQL.
- Evaluating whether a new index improves search speed in a large user table for an e-commerce platform.
- Teaching QA or junior developers how to measure and document database query improvements.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.