Simulate a GIN Index for JSONB in Python
Build a mock Generalized Inverted Index (GIN) that flattens JSON documents into key-value tokens for fast lookup queries, mimicking PostgreSQL JSONB indexing.
Python code
49 linesimport json
import random
from collections import defaultdict
# Mock GIN (Generalized Inverted Index) for JSONB key-value pairs
class GINIndex:
def __init__(self):
self.posting_lists = defaultdict(list) # token -> list of doc_ids
def index(self, doc_id, json_obj):
"""Index a JSON document by flattening all leaf key-value pairs"""
def flatten(obj, prefix=""):
if isinstance(obj, dict):
for k, v in obj.items():
key = f"{prefix}.{k}" if prefix else k
if isinstance(v, (dict, list)):
flatten(v, key)
else:
token = f"{key}={v}"
self.posting_lists[token].append(doc_id)
elif isinstance(obj, list):
for i, v in enumerate(obj):
flatten(v, f"{prefix}[{i}]")
flatten(json_obj)
def search(self, key, value):
"""Search documents by key-value pair"""
token = f"{key}={value}"
return self.posting_lists.get(token, [])
# Demo usage
if __name__ == "__main__":
gin = GINIndex()
docs = [
{"id": 1, "name": "Alice", "age": 30, "address": {"city": "NYC", "zip": 10001}},
{"id": 2, "name": "Bob", "age": 25, "address": {"city": "LA", "zip": 90001}},
{"id": 3, "name": "Charlie", "age": 30, "address": {"city": "NYC", "zip": 10002}},
{"id": 4, "name": "David", "age": 35, "address": {"city": "SF", "zip": 94101}}
]
for doc in docs:
gin.index(doc["id"], doc)
# Query examples
print("People aged 30:", gin.search("age", 30))
print("People in NYC:", gin.search("address.city", "NYC"))
print("People with zip 90001:", gin.search("address.zip", 90001))
Output
People aged 30: [1, 3]
People in NYC: [1, 3]
People with zip 90001: [2]
How it works
This code simulates a GIN index by flattening nested JSON documents into leaf key-value tokens (e.g., 'address.city=NYC') and storing them in inverted posting lists via defaultdict(list). The flatten function recursively handles both dictionaries and lists, building keys like address[0] for arrays. When searching, the exact key-value token is looked up directly in the posting lists, giving O(1) retrieval. This emulates how PostgreSQL GIN indexes accelerate JSONB @> queries and containment searches. The demo indexes four documents and runs three sample queries to show membership lookups across flat and nested fields.
Common mistakes
- Forgetting that arrays need index-based tokens like `tags[0]` instead of just `tags`
- Not normalizing value types — searching with a string `"30"` instead of integer `30` will miss matches
- Overlooking that the mock index doesn't handle wildcard or range queries, only exact token lookups
Variations
- Replace `defaultdict(list)` with `set` to deduplicate document IDs automatically
- Store the index on disk with `sqlite3` or a simple JSON dump for persistence between runs
Real-world use cases
- Prototyping query patterns for a PostgreSQL database where JSONB GIN indexes are planned — validate which queries benefit before migration.
- Building a lightweight in-memory search layer for JSON configuration files in a microservice without adding full-text search infrastructure.
- Teaching or documenting how inverted indexes work internally by implementing a minimal version in pure Python for team training.
Sponsored
More from Database scaling & optimization
- Approximate Count with HyperLogLog in Python medium
- B-Tree Insert and In-Order Traversal in Python hard
- Broadcast a Small Reference Table in Python easy
- Build a Full Text Search Index in Python medium
- Build a Partial Index Mock in Python for Database Filtering easy
- Composite index leftmost prefix in Python medium
Keep learning
Related tutorials and quizzes for this topic.