Python multiprocessing gives you true parallelism: it runs separate Python processes, one per Central Processing Unit (CPU) core, so each escapes the Global Interpreter Lock (GIL) and your heavy math actually runs at the same time. This guide shows when to reach for it over threading, how Pool and ProcessPoolExecutor work, and how to pass data between processes with a queue.

“Concurrency is not parallelism.”

Rob Pike, Go Proverbs

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Advanced | Reading Time: 16 minutes

Here is the decision in one line. If your code is busy crunching numbers and you want it to finish faster, you want multiprocessing. If your code is mostly waiting (for a website, a file, a database), you want threading or asyncio instead. This whole post is about telling those two situations apart, then picking the right tool with confidence.

In the multithreading post you saw the problem. The GIL (the lock that lets only one thread run Python code at a time, covered in depth in the memory management post) means threads cannot speed up heavy math. They take turns. The multiprocessing module gets around this by starting whole separate Python processes. Each process has its own interpreter, its own GIL, and its own slice of memory, so each one can run flat out on a different CPU core. Nothing has to take turns. That is real parallelism.

Think of it like a kitchen. One cook (one process) can only chop so fast. The GIL is the rule that says only one knife may move at a time. Threading hands the single knife back and forth between cooks, so the chopping never actually speeds up. Multiprocessing instead gives each cook their own knife and their own cutting board. Now four cooks really do chop four times the vegetables. The catch is that each cook needs their own board and their own copy of the recipe, which costs time and space to set up. In short, that setup cost is the whole trade-off.

So multiprocessing is not free. Starting a process is much heavier than starting a thread. Because processes do not share memory, any data you pass between them has to be pickled (serialized) and shipped across, which adds overhead. For waiting-heavy (I/O-bound) work, threading or asyncio is simpler and usually faster. For number-crunching (CPU-bound) work where you want every core working, multiprocessing is the answer.

Start at the top and answer one question: what is the task actually doing? If it is busy with heavy computation (math, image processing, parsing), follow the CPU-bound branch to multiprocessing: one process per core, true parallelism. If it is mostly waiting on the network, disk, or an API, you are I/O-bound, and the next question is how many things you wait on at once. A handful (10 to 50) is fine with threading and its shared memory. Thousands of connections call for asyncio and its event loop. Screenshot this tree and pin it; nine times out of ten it picks your tool in seconds.

Prerequisites

Read Multithreading first. It explains the GIL, the exact limitation that Python multiprocessing exists to get around. You also need a multi-core machine to see any speedup, which is basically every laptop made in the last decade. The examples here were run on Python 3.14.6 on an 8-core Windows machine, so your timings and process IDs will differ from the ones shown. That is expected.

Basic Multiprocessing

The cleanest way to start is multiprocessing.Pool. You hand it a function and a list of inputs, and pool.map spreads those inputs across worker processes and collects the results. It works like a teacher splitting a pile of exam papers among four graders: each grader marks their share at the same time, and the papers come back stacked in the original order. Here we run the same heavy job twice: once the plain sequential way, once across four workers. Watch the process IDs (the PID) and the timing.

📄 basic_multiprocess.py: running CPU-bound tasks in parallel

import multiprocessing
import time
import os

def heavy_computation(n):
    """CPU-bound work: add up the squares from 0 to n."""
    result = sum(i * i for i in range(n))
    print(f"PID {os.getpid()}: computed sum for n={n:,}", flush=True)
    return result

if __name__ == "__main__":
    numbers = [10_000_000, 10_000_000, 10_000_000, 10_000_000]

    # One process, one task after another
    start = time.perf_counter()
    results = [heavy_computation(n) for n in numbers]
    print(f"Sequential: {time.perf_counter() - start:.2f}s\n", flush=True)

    # Four worker processes, all running at the same time
    start = time.perf_counter()
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(heavy_computation, numbers)
    print(f"Parallel (4 cores): {time.perf_counter() - start:.2f}s", flush=True)

▶ Output

PID 7136: computed sum for n=10,000,000
PID 7136: computed sum for n=10,000,000
PID 7136: computed sum for n=10,000,000
PID 7136: computed sum for n=10,000,000
Sequential: 6.10s

PID 15020: computed sum for n=10,000,000
PID 6392: computed sum for n=10,000,000
PID 5508: computed sum for n=10,000,000
PID 7012: computed sum for n=10,000,000
Parallel (4 cores): 3.20s

What happened here: Look at the PIDs. In the sequential run, all four lines show the same PID (7136), because one process did everything in order. In the parallel run, four different PIDs show up: four separate processes, each on its own core, each computing at the same time. The job went from about 6.1 seconds to about 3.2 seconds, a little under 2x faster. Why not a clean 4x? Because spawning four fresh Python processes and shipping the inputs and results back and forth costs real time.

That setup tax is fixed, so multiprocessing pays off best on big, long jobs where the computation dwarfs the startup cost. One detail worth noticing: we added flush=True to every print. Without it, the child processes buffer their output and the lines can show up jumbled. flush=True forces each line out immediately so the order matches the code.

ProcessPoolExecutor: The Clean API

Pool is great, but the modern, recommended way to write Python multiprocessing code is ProcessPoolExecutor from the concurrent.futures module. The big win is consistency: its API is identical to ThreadPoolExecutor from the threading post. It is like two rental cars with the same steering wheel and pedals: once you can drive one, you can drive the other without relearning anything. Switching a job from threads to processes is often a one-word change. Same .map(), same .submit(), same with block. Here we check six large numbers for primality across four workers.

📄 pool_executor.py: modern multiprocessing with concurrent.futures

from concurrent.futures import ProcessPoolExecutor
import time

def is_prime(n):
    """Check whether n is prime. Heavy work for very large numbers."""
    if n < 2:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

if __name__ == "__main__":
    candidates = [15485863, 15485866, 32452843, 32452840, 49979687, 67867978]

    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=4) as executor:
        results = list(executor.map(is_prime, candidates))

    for num, prime in zip(candidates, results):
        status = "prime" if prime else "not prime"
        print(f"  {num:>10,} is {status}")
    print(f"\nCompleted in {time.perf_counter() - start:.2f}s")

▶ Output

  15,485,863 is prime
  15,485,866 is not prime
  32,452,843 is prime
  32,452,840 is not prime
  49,979,687 is prime
  67,867,978 is not prime

Completed in 0.33s

What happened here: Two things to notice. First, the results come back in the same order as the input list, even though four workers finished at different moments. executor.map handles that bookkeeping for you, so number one always lines up with answer one. Second, the odd numbers here are genuine primes and the even ones are obviously not (any even number above 2 divides by 2), which makes the output easy to sanity-check at a glance.

The whole batch finished in about a third of a second. If you swap ProcessPoolExecutor for ThreadPoolExecutor on this CPU-bound work, the code still runs, but it gets no faster, because the GIL keeps the threads taking turns. That single-word difference is the entire point of this post.

Sharing Data Between Processes

Threads share memory, so passing data between them is easy. Processes do not, so Python multiprocessing ships proper channels for the job. The most common one is a Queue: a thread-safe, process-safe line that one process puts items into and another takes them out of.

Picture a coffee shop where a barista named Anvi works the machines. Anvi (the producer) makes drinks and sets each one on the pickup counter. Customers (the consumer) grab drinks off that counter, first in, first out. The counter is the queue. Neither side has to know what the other is doing; they just agree on the counter. When Anvi is done for the day, she puts up a “closed” sign so the customers know to stop waiting. In code, that “closed” sign is a sentinel value, here we use None.

📄 shared_data.py: a queue for inter-process communication

from multiprocessing import Process, Queue
import time

def producer(queue, items):
    """Produces items and puts them in the queue."""
    for item in items:
        queue.put(item)
        print(f"Produced: {item}")
    queue.put(None)  # Sentinel to signal done

def consumer(queue, name):
    """Consumes items from the queue."""
    while True:
        item = queue.get()
        if item is None:
            break
        print(f"  {name} consumed: {item}")

if __name__ == "__main__":
    q = Queue()
    data = ["report_jan.csv", "report_feb.csv", "report_mar.csv"]

    p1 = Process(target=producer, args=(q, data))
    p2 = Process(target=consumer, args=(q, "Worker-1"))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

▶ Output

Produced: report_jan.csv
Produced: report_feb.csv
Produced: report_mar.csv
  Worker-1 consumed: report_jan.csv
  Worker-1 consumed: report_feb.csv
  Worker-1 consumed: report_mar.csv

What happened here: Two separate processes, no shared variables, and yet they cooperate cleanly. The producer pushes three filenames onto the queue and then pushes None as the “I am done” signal. The consumer loops forever, pulling one item at a time, until it sees None and breaks out. The producer here happens to finish before the consumer reads, so all three “Produced” lines print first, but that ordering can shift between runs since the two processes run independently.

The data crosses the process boundary safely because the queue pickles each item on the way in and unpickles it on the way out. That also means anything you put on a queue has to be picklable: plain data like strings, numbers, lists, and dicts is fine; open files, database connections, and lambdas are not.

Threading vs Multiprocessing vs Asyncio

Here is the side-by-side that the flowchart is built on. The row that decides almost everything is the first one: what is the task doing? Get that right and the rest follows.

Real-world scenarios

Abstract tables only get you so far. Here is how the choice plays out in jobs you might actually write:

• Resizing 5,000 product photos: multiprocessing. Each resize is pure CPU work, and the images split cleanly across cores. A developer I know, Rahul, cut his nightly image job from one hour to about fifteen minutes on a 4-core box with exactly this change.
• Downloading 40 web pages: threading. The code spends its time waiting for servers to reply, not computing. Threads wait in parallel for almost no setup cost.
• Handling 10,000 live chat connections: asyncio. That many threads or processes would crush your memory; an event loop juggles them all on one thread.
• Training-data feature extraction over a huge CSV (comma-separated values) file: multiprocessing. Heavy number crunching per row, easy to chunk, and you want every core busy.

Decision summary

• Use multiprocessing when the work is CPU-bound (math, image processing, parsing, simulation) and you want to use every core. This is the only one of the three that gives you true parallel computation.
• Use threading when the work is I/O-bound and modest in scale (a few dozen network or disk operations). It is the simplest to write and shares memory for free.
• Use asyncio when the work is I/O-bound and huge in scale (hundreds or thousands of connections at once). One thread, one event loop, the best memory footprint.

Common Mistakes

Mistake 1: forgetting the if __name__ == “__main__” guard

This is the number one multiprocessing bug on Windows and macOS. On those systems, a new process starts by importing your script from scratch. If your pool creation sits at the top level with no guard, every child re-imports the script, hits that pool line again, and spawns more children, which spawn more children. Python catches the loop and raises a RuntimeError in each child instead of melting your laptop. Fair warning: with Pool the parent keeps replacing the dead workers, so the error text below repeats on screen until you stop the script with Ctrl+C.

❌ Wrong: pool created at the top level, no guard

from multiprocessing import Pool

def work(x):
    return x * x

# No guard. Each child re-imports this file and runs Pool(4) again.
with Pool(4) as pool:
    print(pool.map(work, [1, 2, 3, 4]))

▶ Output (trimmed)

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

✅ Correct: wrap the work in the main guard

from multiprocessing import Pool

def work(x):
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        print(pool.map(work, [1, 2, 3, 4]))

Why: the guard means “only run this part when the file is run directly, not when it is imported.” Think of it as a note on a master document that says “only the original acts on this page; photocopies skip it.” Children import the file but their __name__ is not "__main__", so they skip the pool line and just wait for work. Make this your default habit: any script that touches multiprocessing puts the action inside if __name__ == "__main__":.

Mistake 2: using multiprocessing for I/O-bound work

If your task spends its time waiting (downloading pages, reading files, calling an API), spinning up processes is wasted effort. Process startup and pickling overhead can make the parallel version slower than a simple loop, and far slower than threads or asyncio. Reach for multiprocessing only when the CPU is the bottleneck. A quick gut check: open Task Manager (Activity Monitor on macOS, top on Linux) while the slow code runs. If one core is pinned at 100 percent, you are CPU-bound and multiprocessing helps. If the CPU is mostly idle while the program crawls, you are I/O-bound and threading or asyncio is the right call.

Try It Yourself

Write a script that counts the prime numbers between 1 and 10,000,000 using multiprocessing. Split the range into chunks (one per CPU core), have each worker count primes in its chunk, then add up the counts. Time it against a plain single-process version and see how close you get to “number of cores” times faster. You will not hit a perfect multiple, and the gap between your result and the ideal is exactly the process and pickling overhead this post keeps warning about.

Conclusion

You now have the full decision framework: CPU-bound work goes to Python multiprocessing, waiting-heavy work goes to threading or asyncio. You watched the PIDs prove that four processes really do compute at the same time, used ProcessPoolExecutor as the clean modern API, and moved data safely between processes with a Queue and a sentinel. You also met the two mistakes that catch almost everyone: skipping the if __name__ == "__main__" guard and throwing processes at I/O-bound work.

Next comes the third leg of the concurrency stool: Asyncio, async/await for I/O Concurrency, which juggles thousands of waiting tasks on a single thread. And if you want to jump around or revisit earlier topics, browse the full Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Python Multiprocessing

Try each one aloud first. The phrasing you produce under mild pressure is what interviews measure.

Q: What is the difference between the fork, spawn, and forkserver start methods, and what changed in Python 3.14.6?

fork clones the parent process instantly and the child inherits all its memory, but it is unsafe when the parent has running threads and can deadlock. spawn starts a brand new interpreter and imports your module again: slower, but clean, and it is the default on Windows and macOS. forkserver keeps one small, clean server process alive and forks workers from it, combining most of fork’s speed with spawn’s safety. In Python 3.14 the default start method on Linux changed from fork to forkserver precisely because fork plus threads caused hard-to-debug crashes.

Q: Your script runs fine on an older Linux box but crashes on Windows with “Can’t pickle local object” the moment pool.map starts. What is going on?

Windows uses the spawn start method, so the target function and every argument must be pickled and shipped to a fresh worker process. Lambdas, nested functions, and objects holding open connections cannot be pickled, so the transfer fails. Under fork on the older Linux setup, the child simply inherited the parent’s memory, so nothing needed pickling and the bug stayed hidden. The fix is to define the worker function at module level (top of the file) and pass plain, picklable data; the same rule now applies on modern Linux too, since forkserver became the 3.14 default.

Q: You split a CPU-heavy job across 8 workers, but it runs slower than the single-process version and memory usage triples. What do you check first?

First check how much data crosses the process boundary: every argument and result gets pickled, and shipping a large array to each worker can cost more than the computation it enables. Next check task size: thousands of tiny tasks drown in dispatch overhead, so batch them with the chunksize argument to pool.map or executor.map. For big read-only data, load it once per worker with an initializer or put it in multiprocessing.shared_memory instead of copying it into all 8 processes, which is exactly what causes the memory spike.

Q: How do you handle results as soon as each worker finishes instead of waiting in input order?

executor.map and pool.map return results in input order, so one slow first task holds everything else back. With ProcessPoolExecutor, submit each task via executor.submit and loop over concurrent.futures.as_completed, which yields futures the moment they resolve. With Pool, use imap_unordered. This is the standard pattern for progress bars and streaming pipelines.

Q: What happens when a worker process raises an exception, and what does BrokenProcessPool mean?

A normal Python exception in a worker does not vanish: it is pickled, sent back, and re-raised in the parent when you collect that result, whether from pool.map, future.result(), or iterating executor.map. BrokenProcessPool is a different beast: it means a worker died without raising at all, for example a segfault in a C extension, a call to os._exit, or the operating system killing it for eating too much memory. The pool cannot trust its own state after that, so it refuses all further work and you have to find out what killed the worker.

Q: Python 3.14.6 offers an official free-threaded build with no GIL. Does that make multiprocessing obsolete?

No. The free-threaded build does let threads run Python code in parallel, which covers the classic CPU-bound case, but it is a separate opt-in build and parts of the C extension ecosystem are still catching up to it. Multiprocessing also provides things threads never will: full memory isolation, crash containment (one dead worker cannot corrupt the parent), and a natural path to spreading work across multiple machines. On the default GIL build, which is what most production systems run today, multiprocessing remains the way to use every core.

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: Python: Multithreading (Thread, Lock, GIL)

Next: Python: Asyncio, async/await for I/O Concurrency

Series Home: Python + AI/ML Tutorial Series