How Python Handles Concurrency Internally

Let’s be honest — when people first hear that Python has a Global Interpreter Lock (GIL), they often panic. “Does that mean Python can’t do multiple things at once?” Not exactly. The story is more subtle, and understanding it will make you a better Python developer.

The GIL: Python’s Security Guard

Think of the GIL like a security guard standing at the door of a nightclub. Only one person can enter at a time. That’s how Python’s memory management works — only one thread can execute Python bytecode at any given moment.

Why does Python have this? Because CPython’s memory management isn’t thread-safe. Without the GIL, your variables could get corrupted when two threads try to modify them simultaneously. So the GIL protects your data, but it also limits true parallel execution.

Threading vs. Multiprocessing

Here’s where PythonSkillset readers often get confused. Let’s clear it up:

Threading is great for I/O-bound tasks (downloading files, reading databases, waiting for network responses). While one thread waits, another can run. But don’t expect speedups for CPU-heavy work.

Multiprocessing spawns separate Python processes, each with its own GIL. This gives you true parallelism for CPU-intensive tasks like image processing or data analysis.

Example from a real project: At PythonSkillset, we have a web scraper that downloads 50 pages at once using threading. The GIL doesn’t matter here because most time is spent waiting for network responses. But when processing those pages (text analysis), we switch to multiprocessing to utilize all CPU cores.

The asyncio Revolution

Python 3.4 introduced asyncio, which changed everything. Instead of threads, you use coroutines — functions that can pause and resume. Think of it like reading multiple books by switching between them every few minutes.

import asyncio

async def fetch_data(url):
    # Simulating network delay
    await asyncio.sleep(1)
    return f"Data from {url}"

async def main():
    tasks = [fetch_data(f"page_{i}") for i in range(10)]
    results = await asyncio.gather(*tasks)
    return results

asyncio.run(main())

This runs all 10 tasks concurrently in a single thread. The key is await — it tells Python “I’m waiting, do something else.”

What’s Actually Happening Inside

When you write threading.Thread(target=func).start(), Python creates a new thread. But only one thread holds the GIL at a time. The interpreter checks every 100 bytecode instructions (or after a system call) to see if another thread should get the GIL.

With asyncio, there’s no thread switching. The event loop manages which coroutine runs next. This is faster because: - No thread creation overhead - No context switching between threads - No GIL contention

Choosing the Right Tool

For a PythonSkillset developer, here’s your cheat sheet:

• Need to wait for I/O? Use asyncio or threading
• Need math to run fast? Use multiprocessing
• Building a web server? Use asyncio with something like FastAPI
• Processing large datasets? Use multiprocessing with Pool.map()

The Future

Python 3.12 made the GIL optionally removable in certain builds. This is experimental, but it shows the direction. For now, understanding concurrency in Python means knowing when to use each approach.

Remember: Python’s concurrency isn’t broken — it’s just different. Once you understand the guard at the door, you can work with it, not against it.