Parallel Processing with concurrent.futures

Here is the article you requested, written for PythonSkillset.com.

You have a Python script that takes forever to run. It’s downloading data, processing images, or hitting a bunch of APIs one after the other. You know your computer has multiple cores sitting idle. It’s frustrating. You want real parallelism, not just the illusion of it.

That’s where concurrent.futures comes in. It is one of Python’s most user-friendly gateways to truly parallel execution. Unlike threading, which still suffers from Python's Global Interpreter Lock (GIL), concurrent.futures with the ProcessPoolExecutor can bypass the GIL and use multiple CPU cores effectively. It feels like a superpower, and it’s surprisingly easy to set up.

The Core Idea: Submit, Don't Wait

The standard way to run tasks is to loop through them and do them one by one. With concurrent.futures, you submit all your tasks to a pool of workers, and the pool distributes them across available cores or threads.

Think of it like a busy kitchen kitchen. Instead of one chef cooking every dish from start to finish (which is your normal loop), you have a team of chefs. Each one grabs a dish order, cooks it, and plates it. The kitchen manager is the executor. You don’t wait on one dish to finish before the next one starts.

The Two Main Executors

The module gives you two primary tools:

• ThreadPoolExecutor: Great for I/O-bound tasks (file operations, network requests, database queries). These tasks spend most of their time waiting for external data, and the GIL doesn't hurt you much here.
• ProcessPoolExecutor: The one for real parallelism. Use this for CPU-bound tasks (complex math, video processing, heavy data transformation). Each worker gets its own Python interpreter and memory space, completely sidestepping the GIL.

For a tech site like PythonSkillset, the ProcessPoolExecutor is the more powerful tool, so let’s focus on that.

A Concrete Example: Crunching Many Numbers

Imagine you are running a small analytics service. You have to calculate the prime factors of a list of one hundred very large numbers. Doing this one at a time ties up a single core for minutes.

import concurrent.futures
import math
import time

def find_prime_factors(n):
    """A CPU-intensive function to find prime factors."""
    factors = []
    d = 2
    while d * d <= n:
        while n % d == 0:
            factors.append(d)
            n //= d
        d += 1 if d == 2 else 2  # Check only 2 and odd numbers
    if n > 1:
        factors.append(n)
    return factors

# Let's say a list of large numbers arrives from user data
big_numbers = [123456789, 987654321, 111111111, 222222222] * 25 # 100 numbers total

start = time.perf_counter()

# Using ProcessPoolExecutor for real CPU parallelism
with concurrent.futures.ProcessPoolExecutor() as executor:
    # Map the function to the list of arguments
    results = list(executor.map(find_prime_factors, big_numbers))

end = time.perf_counter()
print(f"Parallel processing took {end - start:.2f} seconds.")

The magic happens in two lines: 1. with ... ProcessPoolExecutor() as executor: This creates a pool of workers. By default, it uses as many workers as your computer has CPU cores. 2. executor.map(func, iterable): This applies the function to every element of the list. Crucially, the library handles splitting the work, sending it to different processes, and collecting the results in the correct order.

More Control with submit()

What if you don’t want to wait for the whole map to finish? Imagine you are building a dashboard that needs to show partial results as they come in. You can use submit().

futures = []
for number in big_numbers:
    future = executor.submit(find_prime_factors, number)
    futures.append(future)

# As futures complete, fetch their results
for future in concurrent.futures.as_completed(futures):
    result = future.result()
    # You could log this result, update a progress bar, or send it to a frontend
    print(f"Got a result: {result[:3]}...")  # Just print first few factors

as_completed() gives you the results in the order they finish, not the order you submitted them. This is perfect for live updates or when some tasks are naturally faster than others.

The One Pitfall You Must Know

Processes do not share memory easily. Each worker process starts fresh. Any global variable you set outside the with block will not be visible inside your function unless you pass it as an argument.

Also, if your function is a lambda or a method of a class that isn't picklable, it will fail. Python needs to serialize (pickle) the function to send it to the other processes. Simple, module-level functions work best.

When to Use What

• I/O wait: Downloading 1000 URLs? ThreadPoolExecutor. The bottleneck is the network, not your CPU. Threads are lighter and handle waiting well.
• CPU crunch: Processing 1000 high-res images? ProcessPoolExecutor. You need all the cores you can get.

Wrapping Up

concurrent.futures is not a magic fix for every slow script. It adds overhead to start the processes. If your task only takes 0.001 seconds, the overhead of using a process pool will actually make it slower.

But for any real-world workload where you are doing more than a trivial amount of work, this module will transform your scripts from slow, sequential runners into fast, parallel workers. It is a tool every Python developer at PythonSkillset should have in their belt. Give it a try with your next data-heavy task. You'll be surprised how easy it is to unlock your machine's true potential.