How to Parse JSON Files in Parallel with Python ThreadPoolExecutor
Load and transform JSON records from multiple files concurrently using ThreadPoolExecutor for faster I/O-bound parsing.
Python code
50 linesimport time
from concurrent.futures import ThreadPoolExecutor
import json
def load_json_file(path):
with open(path, 'r') as f:
return json.load(f)
def transform_record(record):
record['full_name'] = f"{record.pop('first_name', '')} {record.pop('last_name', '')}".strip()
record['score'] = int(record.get('score', 0))
return record
def parse_data_parallel(file_paths):
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=4) as executor:
load_futures = [executor.submit(load_json_file, path) for path in file_paths]
raw_data = [f.result() for f in load_futures]
flat_records = [record for file_data in raw_data for record in file_data]
with ThreadPoolExecutor(max_workers=4) as executor:
transformed = list(executor.map(transform_record, flat_records))
elapsed = time.perf_counter() - start
print(f"Parsed {len(transformed)} records across {len(file_paths)} files in {elapsed:.3f}s")
return transformed
def create_sample_files():
import tempfile
import os
data = [
[{"first_name": "Alice", "last_name": "Smith", "score": 95},
{"first_name": "Bob", "last_name": "Jones", "score": 88}],
[{"first_name": "Carol", "last_name": "White", "score": 72},
{"first_name": "Dave", "last_name": "Brown", "score": 91}]
]
paths = []
for i, records in enumerate(data):
path = os.path.join(tempfile.mkdtemp(), f"file_{i}.json")
with open(path, 'w') as f:
json.dump(records, f)
paths.append(path)
return paths
if __name__ == "__main__":
files = create_sample_files()
result = parse_data_parallel(files)
for rec in result:
print(rec)Output
Parsed 4 records across 2 files in 0.001s
{'full_name': 'Alice Smith', 'score': 95}
{'full_name': 'Bob Jones', 'score': 88}
{'full_name': 'Carol White', 'score': 72}
{'full_name': 'Dave Brown', 'score': 91}How it works
ThreadPoolExecutor runs tasks in a thread pool, which suits I/O-bound work like file reading since threads release the GIL during blocking I/O. The first executor submits load_json_file for each path and collects results, while the second maps transform_record over the flattened list. Using executor.map preserves order and handles iteration efficiently. Timing with time.perf_counter gives a high-resolution measure of the concurrent execution.
Common mistakes
- Using threads for CPU-bound transforms — the GIL limits speedup; use processes instead.
- Forgetting to close files or not using context managers, causing resource leaks.
- Assuming `json.load` returns a list — it returns whatever type is in the file, often a dict.
- Hard-coding `max_workers` without considering the number of files or cores.
Variations
- Use `ProcessPoolExecutor` instead of `ThreadPoolExecutor` when transforms are CPU-heavy.
- Use `executor.submit` with `as_completed` to process results as they finish for faster feedback.
Real-world use cases
- Batch-loading multiple config or data files at service startup to reduce cold-start latency.
- Parsing many log files in a data pipeline before aggregating metrics or alerts.
- Fetching and parsing JSON responses from several APIs concurrently to build a single dataset.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.