When to Use Threads vs Processes in Python

Here is the article you requested, written in the tone and style of a PythonSkillset.com contributor.

When to Use Threads vs Processes in Python

You’ve probably written a script that does one thing after another. Then you hit a wall—your program is too slow. The natural instinct is to try and run things in parallel. But in Python, the choice between multithreading and multiprocessing isn’t just a matter of taste. Pick the wrong one, and you could actually make your program slower.

Let’s break this down with a real-world example that any Python developer will recognize.

The Problem with the GIL

First, we need to talk about the elephant in the room: the Global Interpreter Lock (GIL). This is a mutex in CPython (the standard Python interpreter) that prevents multiple native threads from executing Python bytecodes at once. This means that, strictly speaking, your multithreaded Python program is not running in parallel on multiple CPU cores. It’s just switching between threads very quickly.

This sounds terrible, but it’s not a deal-breaker for every job.

Multithreading: Best for I/O-Bound Tasks

Think about a web scraper. Your script sends a request to a server and then sits there, waiting for the response. That waiting time is called I/O (Input/Output). During that wait, the CPU is essentially idle.

Let’s say you work for a company called DataFetch Corp and you need to download product details from 100 different URLs. A simple loop would work, but you’d spend most of the time just waiting for the network.

With multithreading, when one thread is waiting for data to come back from the network, the GIL is released. Another thread can then jump in and make its own request. This gives the illusion of parallelism, but more importantly, it keeps the CPU busy while the network catches up.

Here’s a minimal example. We’ll use concurrent.futures because it’s cleaner than managing threads manually.

import threading
import time
from concurrent.futures import ThreadPoolExecutor

def fetch_data(url):
    # Simulate a network request with a sleep
    time.sleep(2)
    return f"Data from {url}"

urls = [f"https://api.example.com/item/{i}" for i in range(10)]

start = time.time()
with ThreadPoolExecutor(max_workers=5) as executor:
    results = executor.map(fetch_data, urls)
end = time.time()

print(f"Multithreading took {end - start:.2f} seconds")

If you ran this sequentially, it would take 20 seconds (2 seconds * 10). With 5 threads, it took about 4 seconds. This is the sweet spot for threading.

Multiprocessing: Best for CPU-Bound Tasks

Now, imagine you’re working at PhotoLab Inc and your job is to apply a complex filter to 10,000 high-resolution images. This isn’t waiting for a network. This is pure number crunching. The CPU is working at 100%, and the GIL becomes a bottleneck.

In this scenario, threading is useless because the GIL prevents any real parallel execution. Each thread would have to wait for the other to finish its CPU cycle.

Multiprocessing bypasses this by spawning separate Python processes. Each process has its own Python interpreter and its own memory space. This means you can truly use all your CPU cores.

Let’s look at the same function, but this time it’s doing CPU work.

import time
from concurrent.futures import ProcessPoolExecutor
import math

def complex_calculation(x):
    # Simulate a heavy CPU task
    for _ in range(100_000_000):
        math.sqrt(x)
    return x * x

numbers = range(8)  # Let's say you have 8 CPU cores

start = time.time()
# Note the change from ThreadPoolExecutor to ProcessPoolExecutor
with ProcessPoolExecutor() as executor:
    results = executor.map(complex_calculation, numbers)
end = time.time()

print(f"Multiprocessing took {end - start:.2f} seconds")

If you ran this sequentially, it would be slow. If you used threads, it would be virtually no faster than sequential because of the GIL. But with a ProcessPoolExecutor, it splits the work across your CPU cores, and you see a near-linear speedup.

The Trade-off You Need to Know

Multiprocessing isn’t a free lunch. Spawning a process is much more expensive than spawning a thread. It takes more memory and time to start. Also, sharing data between processes is harder. You can’t just use a global variable. You have to use special objects like a multiprocessing.Queue or a multiprocessing.Value.

Threads, on the other hand, share the same memory space. It’s easy to pass data around. But you have to be careful with race conditions where two threads try to modify the same variable at the same time.

A Simple Decision Tree

Next time you’re at your desk at PythonSkillset and you need to speed something up, ask yourself these two questions:

1. Is my program waiting for the network, a file read, or a database query? If yes, go with ThreadPoolExecutor from concurrent.futures. It’s simple and effective for I/O-bound work.
2. Is my program doing heavy math, image processing, or data parsing where the CPU is the bottleneck? If yes, use ProcessPoolExecutor. The overhead of creating processes is worth the massive speed gain.

You could also look at the asyncio library, which is another way to handle I/O-bound tasks without threads. But for the classic choice between threads and processes, this rule of thumb has served me well for years.

The GIL is a limitation, but it’s not a trap. It just forces you to think clearly about what your program is actually doing. And that’s a good habit for any developer.