Why Is Pool.map Slower Than Normal Map?

I'm trying the following code: import multiprocessing import time import random def square(x): return x**2 pool = multiprocessing.Pool(4) l = [random.random() for i in xrang

Solution 1:

pool.map splits a list into N jobs (where N is the size of the list) and dispatches those to the processes.

The work a single process is doing is shown in your code:

def square(x):
    return x**2

This operation takes very little time on modern CPUs, no matter how big the number is.

In your example you're creating a huge list and performing an irrelevant operation on every single element. Of course the IPC overhead will be greater compared to the regular map function which is optimized for fast looping.

In order to see your example working as you expect, just add a time.sleep(0.1) call to the square function. This simulates a long running task. Of course you might want to reduce the size of the list or it will take forever to complete.