How to Speed Up Downloads with ThreadPoolExecutor in Python

Compare sequential and thread-pool download loops to measure real speedup when I/O s bound.

Medium Python 3.9+ Aug 9, 2026 Concurrency & performance 13 views 0 copies

Python code

35 lines
Python 3.9+
import time
import threading
from concurrent.futures import ThreadPoolExecutor

def download_file(file_id):
    """Simulate fetching a file by sleeping briefly."""
    time.sleep(0.2)  # pretend network latency
    return f"file_{file_id}"

def sequential_downloads(num_files):
    """Process files one at a time."""
    start = time.perf_counter()
    results = [download_file(i) for i in range(num_files)]
    elapsed = time.perf_counter() - start
    return results, elapsed

def concurrent_downloads(num_files):
    """Process files in parallel using a thread pool."""
    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=min(num_files, 4)) as executor:
        results = list(executor.map(download_file, range(num_files)))
    elapsed = time.perf_counter() - start
    return results, elapsed

if __name__ == "__main__":
    file_count = 8

    seq_results, seq_time = sequential_downloads(file_count)
    print(f"Sequential: {seq_time:.3f}s — files: {seq_results}")

    con_results, con_time = concurrent_downloads(file_count)
    print(f"Concurrent: {con_time:.3f}s — files: {con_results}")

    speedup = seq_time / con_time if con_time > 0 else float("inf")
    print(f"Speedup factor: {speedup:.2f}x")

Output

stdout
Sequential: 1.602s — files: ['file_0', 'file_1', 'file_2', 'file_3', 'file_4', 'file_5', 'file_6', 'file_7']
Concurrent: 0.403s — files: ['file_0', 'file_1', 'file_2', 'file_3', 'file_4', 'file_5', 'file_6', 'file_7']
Speedup factor: 3.98x

How it works

time.sleep(0.2) simulates network I/O, which releases the GIL, allowing threads to run in parallel. ThreadPoolExecutor creates a pool of up to 4 worker threads that pull tasks from an internal queue, so 8 downloads finish in roughly the time of 2 sequential ones. The executor.map preserves input order while returning results as they complete, making it easy to pair results with file IDs. Timing with perf_counter gives a high-resolution measure that excludes overhead outside the block. The real speedup depends on the ratio of I/O wait time to CPU work; CPU-bound tasks would benefit from processes instead.

Common mistakes

  • Using threads for CPU-heavy work where the GIL serializes execution, giving no speedup.
  • Setting max_workers larger than the number of concurrent I/O operations the system can handle.
  • Timing the entire list comprehension without capturing the executor context, which includes shutdown time.
  • Assuming a linear speedup equal to the number of threads without accounting for overhead.

Variations

  1. Use `asyncio` with `aiohttp` for fully async HTTP downloads without threads.
  2. Use `ProcessPoolExecutor` for CPU-bound tasks that need parallel computation across cores.

Real-world use cases

  • Batch-downloading images or assets from a CDN for a data pipeline, cutting total fetch time.
  • Fetching multiple API endpoints concurrently in a web scraper to gather data for research.
  • Running parallel checks against multiple services during a health-monitoring script.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Concurrency & performance

Related tutorials and quizzes for this topic.