Simulate a GIN Index for JSONB in Python

Build a mock Generalized Inverted Index (GIN) that flattens JSON documents into key-value tokens for fast lookup queries, mimicking PostgreSQL JSONB indexing.

Medium Python 3.9+ Aug 9, 2026 Database scaling & optimization 14 views 0 copies

Python code

49 lines
Python 3.9+
import json
import random
from collections import defaultdict

# Mock GIN (Generalized Inverted Index) for JSONB key-value pairs
class GINIndex:
    def __init__(self):
        self.posting_lists = defaultdict(list)  # token -> list of doc_ids
    
    def index(self, doc_id, json_obj):
        """Index a JSON document by flattening all leaf key-value pairs"""
        def flatten(obj, prefix=""):
            if isinstance(obj, dict):
                for k, v in obj.items():
                    key = f"{prefix}.{k}" if prefix else k
                    if isinstance(v, (dict, list)):
                        flatten(v, key)
                    else:
                        token = f"{key}={v}"
                        self.posting_lists[token].append(doc_id)
            elif isinstance(obj, list):
                for i, v in enumerate(obj):
                    flatten(v, f"{prefix}[{i}]")
        
        flatten(json_obj)
    
    def search(self, key, value):
        """Search documents by key-value pair"""
        token = f"{key}={value}"
        return self.posting_lists.get(token, [])

# Demo usage
if __name__ == "__main__":
    gin = GINIndex()
    
    docs = [
        {"id": 1, "name": "Alice", "age": 30, "address": {"city": "NYC", "zip": 10001}},
        {"id": 2, "name": "Bob", "age": 25, "address": {"city": "LA", "zip": 90001}},
        {"id": 3, "name": "Charlie", "age": 30, "address": {"city": "NYC", "zip": 10002}},
        {"id": 4, "name": "David", "age": 35, "address": {"city": "SF", "zip": 94101}}
    ]
    
    for doc in docs:
        gin.index(doc["id"], doc)
    
    # Query examples
    print("People aged 30:", gin.search("age", 30))
    print("People in NYC:", gin.search("address.city", "NYC"))
    print("People with zip 90001:", gin.search("address.zip", 90001))

Output

stdout
People aged 30: [1, 3]
People in NYC: [1, 3]
People with zip 90001: [2]

How it works

This code simulates a GIN index by flattening nested JSON documents into leaf key-value tokens (e.g., 'address.city=NYC') and storing them in inverted posting lists via defaultdict(list). The flatten function recursively handles both dictionaries and lists, building keys like address[0] for arrays. When searching, the exact key-value token is looked up directly in the posting lists, giving O(1) retrieval. This emulates how PostgreSQL GIN indexes accelerate JSONB @> queries and containment searches. The demo indexes four documents and runs three sample queries to show membership lookups across flat and nested fields.

Common mistakes

  • Forgetting that arrays need index-based tokens like `tags[0]` instead of just `tags`
  • Not normalizing value types — searching with a string `"30"` instead of integer `30` will miss matches
  • Overlooking that the mock index doesn't handle wildcard or range queries, only exact token lookups

Variations

  1. Replace `defaultdict(list)` with `set` to deduplicate document IDs automatically
  2. Store the index on disk with `sqlite3` or a simple JSON dump for persistence between runs

Real-world use cases

  • Prototyping query patterns for a PostgreSQL database where JSONB GIN indexes are planned — validate which queries benefit before migration.
  • Building a lightweight in-memory search layer for JSON configuration files in a microservice without adding full-text search infrastructure.
  • Teaching or documenting how inverted indexes work internally by implementing a minimal version in pure Python for team training.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Related tutorials and quizzes for this topic.