3

I'm following the answer on Is there a simple process-based parallel map for python? using Python 3.6.1. I'm trying to use it the same exact way as I would use the regular map function. Why does this happen? How can I create a parallel map function for use with lambda? 

def map_parallel(func, iterable, n_jobs=-1):
    if n_jobs == -1:
        n_jobs = multiprocessing.cpu_count()
    pool = multiprocessing.Pool(processes=n_jobs)
    return pool.map(func, iterable)

map_parallel(lambda x: x + 3, range(100))


PicklingError: Can't pickle <function <lambda> at 0x185284f28>: attribute lookup <lambda> on __main__ failed
Community
  • 1
  • 1
O.rka
  • 24,289
  • 52
  • 152
  • 253

1 Answers1

0

multiprocessing pickle objects to pass between processes. lambda is not serializable (See What can be pickled and unpickled?); it can't be passed between processes.

You need to define a function, and pass that.

def add3(n):
    return n + 3

map_parallel(add3, range(100))

Or for this case, you can use functools.partial with operator.add:

import functools
import operator

map_parallel(functools.partial(operator.add, 3), range(100))
falsetru
  • 314,667
  • 49
  • 610
  • 551
  • Is it possible to convert `lambda` to a `function`? I rarely ever use `map` with `functions` and use it on the fly with transforming iterables – O.rka May 04 '17 at 01:54
  • @O.rka, Check [this answer](http://stackoverflow.com/a/25353243/2225682). – falsetru May 04 '17 at 01:56