How to Speed Up Data Filtering with Python ThreadPoolExecutor
This code compares sequential filtering of even numbers with a threaded version using ThreadPoolExecutor, showing a measurable speedup for I/O-bound work.
Python code
37 linesimport time
from concurrent.futures import ThreadPoolExecutor
import random
def is_even(number):
time.sleep(0.001) # simulate work
return number % 2 == 0
def filter_even_sequential(numbers):
return [n for n in numbers if is_even(n)]
def filter_even_threaded(numbers):
with ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(is_even, numbers))
return [n for n, keep in zip(numbers, results) if keep]
if __name__ == "__main__":
random.seed(42)
data = [random.randint(1, 100) for _ in range(100)]
start = time.perf_counter()
sequential_result = filter_even_sequential(data)
sequential_time = time.perf_counter() - start
start = time.perf_counter()
threaded_result = filter_even_threaded(data)
threaded_time = time.perf_counter() - start
assert sequential_result == threaded_result
print(f"Sequential: {sequential_time:.4f} seconds, {len(sequential_result)} evens")
print(f"Threaded: {threaded_time:.4f} seconds, {len(threaded_result)} evens")
print(f"Speedup: {sequential_time / threaded_time:.2f}x")Output
Sequential: 0.1140 seconds, 58 evens
Threaded: 0.0310 seconds, 58 evens
Speedup: 3.68xHow it works
The ThreadPoolExecutor creates a pool of worker threads that process items concurrently. Using executor.map, each call to is_even runs in a separate thread, overlapping the simulated I/O wait. The zip pairs original numbers with boolean results to filter correctly. For I/O-bound tasks, threads provide speedup because the Global Interpreter Lock (GIL) releases during blocking calls. This pattern is simple and safe for pure functions without shared state.
Common mistakes
- Using threads for CPU-bound tasks where the GIL limits parallelism.
- Forgetting that `executor.map` returns results in order, which is fine here but not needed if order doesn't matter.
- Creating a new ThreadPoolExecutor per call when you could reuse it for multiple tasks.
Variations
- Use `concurrent.futures.ProcessPoolExecutor` for CPU-bound operations that need true parallelism.
- Replace `executor.map` with `executor.submit` and `as_completed` for more control over scheduling.
Real-world use cases
- Fetching data from multiple APIs concurrently and filtering responses before aggregation.
- Processing a batch of image files (e.g., resizing with blocking I/O) in parallel for faster bulk operations.
- Running database queries in parallel for a data pipeline, filtering results as they arrive.
Sponsored
More from Concurrency & performance
- Benchmark list.append vs deque.append in Python medium
- Build a Python Performance Profiler That Generates Readable Reports medium
- Graceful Shutdown Executor Context Manager in Python medium
- How to Build a Producer-Consumer Pattern with asyncio.Queue in Python medium
- How to Cancel an asyncio Task with Graceful Cleanup in Python medium
- How to Convert Data in Parallel with ThreadPoolExecutor in Python easy
Keep learning
Related tutorials and quizzes for this topic.