Parallel Extract Multiple Sources with Threads in Python

Extract data from multiple sources in parallel using ThreadPoolExecutor and verify results match sequential processing.

Easy Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

Python code

25 lines
Python 3.9+
import threading
from concurrent.futures import ThreadPoolExecutor

def extract_from_source(source):
    """Simulate extracting data from a source."""
    return f"Data from {source}"

def main():
    sources = ["source_a", "source_b", "source_c", "source_d"]
    
    # Sequential extraction for comparison
    sequential_results = [extract_from_source(s) for s in sources]
    print("Sequential results:", sequential_results)
    
    # Parallel extraction using ThreadPoolExecutor
    with ThreadPoolExecutor(max_workers=4) as executor:
        parallel_results = list(executor.map(extract_from_source, sources))
    print("Parallel results:  ", parallel_results)

    # Verify both approaches produce the same output
    assert sequential_results == parallel_results, "Mismatch between sequential and parallel results"
    print("Both approaches produced identical results.")

if __name__ == "__main__":
    main()

Output

stdout
Sequential results: ['Data from source_a', 'Data from source_b', 'Data from source_c', 'Data from source_d']
Parallel results:   ['Data from source_a', 'Data from source_b', 'Data from source_c', 'Data from source_d']
Both approaches produced identical results.

How it works

ThreadPoolExecutor creates a pool of worker threads (here 4) and distributes the sources list across them via executor.map. Each thread runs extract_from_source independently, and results are collected in the same order as input. This is safe for I/O-bound tasks because threads can overlap while waiting on I/O. The with block ensures threads are cleaned up automatically. Sequential vs parallel results match, confirming the parallel extraction is correct.

Common mistakes

  • Using ThreadPoolExecutor for CPU-bound tasks where GIL limits speedup.
  • Forgetting that `executor.map` returns results in input order, which may not be completion order.
  • Sharing mutable state across threads without locks, causing race conditions.

Variations

  1. Use `as_completed` if you need results in completion order.
  2. Use `process_pool_executor` for CPU-bound extraction tasks.

Real-world use cases

  • Fetching data from multiple REST APIs concurrently to reduce total latency.
  • Reading multiple database shards or tables in parallel for ETL jobs.
  • Scraping several web pages at once in a data collection pipeline.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.