How to Batch Load JSON Data in Python for Database Optimization

This code parses JSON data into records and loads them in batches to simulate efficient database insertion, reducing load and improving performance.

Easy Python 3.9+ Aug 9, 2026 Database scaling & optimization 12 views 0 copies

json batching database optimization data-loading

Python code

29 lines
Python 3.9+
Run it 
import json
import time

def parse_and_load(data, batch_size=100):
    """
    Parse JSON data and batch-load into a list of dicts.
    Demonstrates batching for database efficiency.
    """
    records = json.loads(data)
    batches = []

    for i in range(0, len(records), batch_size):
        batch = records[i:i + batch_size]
        batches.append(batch)
        print(f"Batch {len(batches)}: {len(batch)} rows")
        time.sleep(0.01)  # Simulate network/storage latency

    return batches

def summary(batches):
    total = sum(len(b) for b in batches)
    print(f"Total records: {total}")
    print(f"Total batches: {len(batches)}")
    print(f"Average batch size: {total / len(batches):.1f}")

if __name__ == "__main__":
    sample_data = json.dumps([{"id": i, "value": i * 2} for i in range(250)])
    result = parse_and_load(sample_data, batch_size=100)
    summary(result)

Output

stdout 
Batch 1: 100 rows
Batch 2: 100 rows
Batch 3: 50 rows
Total records: 250
Total batches: 3
Average batch size: 83.3

How it works

The json.loads function converts JSON strings into Python objects, here a list of dicts. Slicing with records[i:i + batch_size] creates batches of the specified size, allowing the database to process data in chunks rather than all at once, which reduces memory usage and network overhead. A time.sleep simulates latency, making the batching pattern visible. The summary function calculates totals and averages, useful for monitoring load progress. This approach is a common pattern for optimizing database inserts in production environments.

Common mistakes

  • Forgetting to handle cases where the record count is not a multiple of batch size
  • Not using a loop to process batches one by one and instead loading everything at once
  • Ignoring the need to close database connections or commit transactions after each batch

Variations

  1. Use `executemany` or `bulk_insert` methods in database libraries like SQLAlchemy for faster batch inserts
  2. Generate batches lazily with a generator expression to save memory when dealing with huge datasets

Real-world use cases

  • Bulk inserting user data from an API into a PostgreSQL table in chunks to avoid locking issues.
  • Processing large CSV exports in batches to keep memory usage low on a data pipeline.
  • Loading IoT sensor readings into a time-series database with controlled batch sizes for throughput.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Database scaling & optimization

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes