multiprocessing vs threading

Both run concurrent work, but they use different OS primitives and interact with the GIL differently.

Threading (threading)

• Multiple threads in one process, shared memory.
• Lightweight to start; great for I/O waits.
• Limited CPU parallelism in CPython because of the GIL.
• Shared data needs locks/queues to avoid races.

Multiprocessing (multiprocessing)

• Multiple processes, each with its own Python interpreter and GIL.
• True multi-core CPU parallelism for Python code.
• Higher startup/memory cost; share data via pickling, queues, managers, shared memory.
• Harder debugging; must protect __main__ on some platforms (if __name__ == "__main__":).

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def cpu_heavy(n):
    return sum(i * i for i in range(n))

def io_heavy(url_chunk):
    # pretend to fetch / sleep on I/O
    return len(url_chunk)

# I/O-bound → threads often enough
with ThreadPoolExecutor(max_workers=8) as pool:
    list(pool.map(io_heavy, ["a", "b", "c"]))

# CPU-bound → processes
if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as pool:
        print(list(pool.map(cpu_heavy, [10**5] * 4)))

Also consider: asyncio for many concurrent I/O tasks on one thread with cooperative multitasking.

Decision guide: I/O wait → threads/asyncio; CPU crunch in Python → processes.