How to Speed Up Downloads with ThreadPoolExecutor in Python
Compare sequential and thread-pool download loops to measure real speedup when I/O s bound.
Python code
35 linesimport time
import threading
from concurrent.futures import ThreadPoolExecutor
def download_file(file_id):
"""Simulate fetching a file by sleeping briefly."""
time.sleep(0.2) # pretend network latency
return f"file_{file_id}"
def sequential_downloads(num_files):
"""Process files one at a time."""
start = time.perf_counter()
results = [download_file(i) for i in range(num_files)]
elapsed = time.perf_counter() - start
return results, elapsed
def concurrent_downloads(num_files):
"""Process files in parallel using a thread pool."""
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=min(num_files, 4)) as executor:
results = list(executor.map(download_file, range(num_files)))
elapsed = time.perf_counter() - start
return results, elapsed
if __name__ == "__main__":
file_count = 8
seq_results, seq_time = sequential_downloads(file_count)
print(f"Sequential: {seq_time:.3f}s — files: {seq_results}")
con_results, con_time = concurrent_downloads(file_count)
print(f"Concurrent: {con_time:.3f}s — files: {con_results}")
speedup = seq_time / con_time if con_time > 0 else float("inf")
print(f"Speedup factor: {speedup:.2f}x")
Output
Sequential: 1.602s — files: ['file_0', 'file_1', 'file_2', 'file_3', 'file_4', 'file_5', 'file_6', 'file_7']
Concurrent: 0.403s — files: ['file_0', 'file_1', 'file_2', 'file_3', 'file_4', 'file_5', 'file_6', 'file_7']
Speedup factor: 3.98x
How it works
time.sleep(0.2) simulates network I/O, which releases the GIL, allowing threads to run in parallel. ThreadPoolExecutor creates a pool of up to 4 worker threads that pull tasks from an internal queue, so 8 downloads finish in roughly the time of 2 sequential ones. The executor.map preserves input order while returning results as they complete, making it easy to pair results with file IDs. Timing with perf_counter gives a high-resolution measure that excludes overhead outside the block. The real speedup depends on the ratio of I/O wait time to CPU work; CPU-bound tasks would benefit from processes instead.
Common mistakes
- Using threads for CPU-heavy work where the GIL serializes execution, giving no speedup.
- Setting max_workers larger than the number of concurrent I/O operations the system can handle.
- Timing the entire list comprehension without capturing the executor context, which includes shutdown time.
- Assuming a linear speedup equal to the number of threads without accounting for overhead.
Variations
- Use `asyncio` with `aiohttp` for fully async HTTP downloads without threads.
- Use `ProcessPoolExecutor` for CPU-bound tasks that need parallel computation across cores.
Real-world use cases
- Batch-downloading images or assets from a CDN for a data pipeline, cutting total fetch time.
- Fetching multiple API endpoints concurrently in a web scraper to gather data for research.
- Running parallel checks against multiple services during a health-monitoring script.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.