Python, parallelization with joblib: Delayed with multiple arguments

Question

I am using something similar to the following to parallelize a for loop over two matrices

from joblib import Parallel, delayed
import numpy

def processInput(i,j):
    for k in range(len(i)):
        i[k] = 1
    for t in range(len(b)):
        j[t] = 0
return i,j

a = numpy.eye(3)
b = numpy.eye(3)

num_cores = 2
(a,b) = Parallel(n_jobs=num_cores)(delayed(processInput)(i,j) for i,j in zip(a,b))

but I'm getting the following error: Too many values to unpack (expected 2)

Is there a way to return 2 values with delayed? Or what solution would you propose?

Also, a bit OP, is there a more compact way, like the following (which doesn't actually modify anything) to process the matrices?

from joblib import Parallel, delayed
def processInput(i,j):
    for k in i:
        k = 1
    for t in b:
        t = 0
return i,j

I would like to avoid the use of has_shareable_memory anyway, to avoid possible bad interactions in the actual script and lower performances(?)

score 15 · Accepted Answer · answered Dec 09 '16 at 20:23

Probably too late, but as an answer to the first part of your question: Just return a tuple in your delayed function.

return (i,j)

And for the variable holding the output of all your delayed functions

results = Parallel(n_jobs=num_cores)(delayed(processInput)(i,j) for i,j in zip(a,b))

Now results is a list of tuples each holding some (i,j) and you can just iterate through results.

Python, parallelization with joblib: Delayed with multiple arguments

1 Answers1