What is the best way to parallelize a for loop in python (2020)?

Question

This is not a duplicate question. I came across this question. And did not find my answer.

I want to parallelize a for loop in python. By parallelization I meant that:

Every iteration of the loop runs independently and not sequentially (Not like the whole for loop separate from the main program but for loop still sequential)
Solution should be cross-platform i.e. works on all platforms. Like When I was trying out joblib, I could not make it work on Ubuntu or Ray which does not work on Windows

Parent question is really old and I was thinking if there is a better way to do it now?

Right. And either the built in `multiprocessing` or newer, slightly higher level `concurrent.futures` as detailed in various answers addresses your question — juanpa.arrivillaga, Jun 16 '20 at 20:57
@juanpa.arrivillaga I mentioned that it has some issues and it does not answer my question! Could we not close it so early so that someone with better way to achieve this can answer it — , Jun 16 '20 at 20:58
No, those are your options, there isn't anything new, and many of the answers in that thread have been added in recent years. This is my opinion. You are free to vote to reopen — juanpa.arrivillaga, Jun 16 '20 at 21:00
I do not have enough repo and you have already closed it. It will not be reopened util someone who read the other question and read this one and thinks that I am right. Until then I could not get answers from the people who might have new apporaches — , Jun 16 '20 at 21:05
It will not be reopened, ever, because you have a misunderstanding of how Stack Overflow works. On Stack Overflow, old questions are expected to be updated as time passes, not pointlessly re-asked. This is what distinguishes the Stack Exchange Network from a forum, and ensures it remains helpful to future searchers. — pppery, Jun 17 '20 at 00:24

Amin Guermazi · Answer 1 · 2020-06-16T20:24:09.993

-1

You could use Thread objects.

from threading import Thread
from time import sleep
from random import randint

def f(i):
    sleep(randint(1, 3))
    print(f'function called from Thread {i}')

for i in range(5):
    Thread(target=f, args=(i,)).start()

and output

function called from Thread 4
function called from Thread 3
function called from Thread 1
function called from Thread 0
function called from Thread 2

edited Jun 16 '20 at 20:24

answered Jun 16 '20 at 20:18

Amin Guermazi

1,372
5
16

Could you please exlpain why I got a downvote! – Amin Guermazi Jun 16 '20 at 20:28
1

threads are not parallel on this level in python – thebjorn Jun 16 '20 at 20:29
what do you mean by 'on this level'? – Amin Guermazi Jun 16 '20 at 20:31
the GIL makes threaded code run in parallel, but only one thread at a time - you'll need a combination of subprocesses and threads to achieve parallellism (check the link in the question for details on how to do this). The overhead is also enormous... – thebjorn Jun 16 '20 at 20:33
Only one thread can run at a time in CPython due to the GIL. – juanpa.arrivillaga Jun 16 '20 at 20:38
The question doesn't specify that we must use CPython – Amin Guermazi Jun 16 '20 at 20:41
your answer doesn't specify that you're not, and that comment is just pedantic without even adding humor -- I'm sure you can do better. – thebjorn Jun 16 '20 at 20:50

Hamza · Answer 2 · 2020-06-17T11:46:55.637

This is the easiest way to do it in 2020! (It runs every iteration independently & this works on both ubuntu and Windows)

You can use asyncio. (Documentation can be found here). It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem.

import asyncio

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def your_function(argument):
    #code

Now this function will be run in parallel whenever called without putting main program into wait state. You can use it to parallelize for loop as well. When called for a for loop, though loop is sequential but every iteration runs in parallel to the main program as soon as interpreter gets there. For instance:

@background
def your_function(argument):
    time.sleep(5)
    print('function finished for '+str(argument))


for i in range(10):
    your_function(i)


print('loop finished')

This produces following output:

loop finished
function finished for 4
function finished for 8
function finished for 0
function finished for 3
function finished for 6
function finished for 2
function finished for 5
function finished for 7
function finished for 9
function finished for 1

But @juanpa.arrivillaga asynchronous will also run every iteration independently. Though I agree its not parallel — Hamza, Jun 17 '20 at 11:48
It doesn't solve the initial question, but you helped me a lot without much of boilerplate code. It would be brilliant if the answer also had code to deal with waiting for all these async tasks, but I figured it myself with some quick and dirty `while` loop — Alexey Larionov, Oct 18 '20 at 20:29

What is the best way to parallelize a for loop in python (2020)?

2 Answers2