How to Convert Data in Parallel with ThreadPoolExecutor in Python

This example demonstrates converting a list of items in parallel using ThreadPoolExecutor, showing performance gains over serial processing.

Easy Python 3.9+ Aug 9, 2026 Concurrency & performance 15 views 0 copies

concurrency threadpoolexecutor parallelism performance io-bound

Python code

25 lines

Python 3.9+

import time
from concurrent.futures import ThreadPoolExecutor


def convert_data(item):
    """Simulate a CPU/IO-bound conversion task."""
    time.sleep(0.05)  # simulate work
    return item.upper()


if __name__ == "__main__":
    items = [f"item_{i}" for i in range(20)]

    start = time.perf_counter()
    serial_results = [convert_data(item) for item in items]
    serial_time = time.perf_counter() - start

    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=4) as executor:
        parallel_results = list(executor.map(convert_data, items))
    parallel_time = time.perf_counter() - start

    print(f"Serial:   {serial_time:.3f}s -> {serial_results[:3]}...")
    print(f"Parallel: {parallel_time:.3f}s -> {parallel_results[:3]}...")
    print(f"Speedup:  {serial_time / parallel_time:.2f}x")

Output

stdout

Serial:   1.030s -> ['ITEM_0', 'ITEM_1', 'ITEM_2']...
Parallel: 0.270s -> ['ITEM_0', 'ITEM_1', 'ITEM_2']...
Speedup:  3.81x

How it works

The code uses time.sleep to simulate work that is IO-bound, which is ideal for threads because they release the GIL during blocking calls. ThreadPoolExecutor.map applies the function to each item in order and returns results in the same order as the input. Using perf_counter gives high-resolution timing to measure the speedup. The parallel version starts multiple threads, so the total time is much lower than the serial version, especially with many items.

Common mistakes

Using threads for CPU-bound tasks like heavy math, which doesn't speed up due to the GIL
Forgetting to use `list()` around `executor.map` to consume the lazy iterator and collect results
Assuming `executor.map` returns results in a different order than input; it preserves order

Variations

Use `executor.submit` with `as_completed` if you need results as they finish rather than preserving order.
For CPU-bound tasks, switch to `ProcessPoolExecutor` to bypass the GIL and achieve real parallelism.

Real-world use cases

Batch-converting file names to uppercase in a data pipeline where each operation involves I/O like reading metadata.
Parallelizing network requests to multiple APIs when normalizing response data into a common format.
Processing a queue of images to apply a filter that spends time waiting on disk I/O, improving throughput.

How to Convert Data in Parallel with ThreadPoolExecutor in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Concurrency & performance

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Concurrency & performance

Keep learning

Tutorials

Quizzes