Parallel Extract Multiple Sources with Threads in Python
Extract data from multiple sources in parallel using ThreadPoolExecutor and verify results match sequential processing.
Python code
25 linesimport threading
from concurrent.futures import ThreadPoolExecutor
def extract_from_source(source):
"""Simulate extracting data from a source."""
return f"Data from {source}"
def main():
sources = ["source_a", "source_b", "source_c", "source_d"]
# Sequential extraction for comparison
sequential_results = [extract_from_source(s) for s in sources]
print("Sequential results:", sequential_results)
# Parallel extraction using ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=4) as executor:
parallel_results = list(executor.map(extract_from_source, sources))
print("Parallel results: ", parallel_results)
# Verify both approaches produce the same output
assert sequential_results == parallel_results, "Mismatch between sequential and parallel results"
print("Both approaches produced identical results.")
if __name__ == "__main__":
main()
Output
Sequential results: ['Data from source_a', 'Data from source_b', 'Data from source_c', 'Data from source_d']
Parallel results: ['Data from source_a', 'Data from source_b', 'Data from source_c', 'Data from source_d']
Both approaches produced identical results.
How it works
ThreadPoolExecutor creates a pool of worker threads (here 4) and distributes the sources list across them via executor.map. Each thread runs extract_from_source independently, and results are collected in the same order as input. This is safe for I/O-bound tasks because threads can overlap while waiting on I/O. The with block ensures threads are cleaned up automatically. Sequential vs parallel results match, confirming the parallel extraction is correct.
Common mistakes
- Using ThreadPoolExecutor for CPU-bound tasks where GIL limits speedup.
- Forgetting that `executor.map` returns results in input order, which may not be completion order.
- Sharing mutable state across threads without locks, causing race conditions.
Variations
- Use `as_completed` if you need results in completion order.
- Use `process_pool_executor` for CPU-bound extraction tasks.
Real-world use cases
- Fetching data from multiple REST APIs concurrently to reduce total latency.
- Reading multiple database shards or tables in parallel for ETL jobs.
- Scraping several web pages at once in a data collection pipeline.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.