Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

696 questions
12
votes
1 answer

Can't pickle coroutine objects when ProcessPoolExecutor is used in class

I'm trying to get asyncio work with subprocesses and limitations. I've accomplish this in functional way, but when I tried to implement same logic in opp style several problems showd up. Mostly Can't pickle coroutine/generator errors. I tracked some…
Alex Smith
  • 192
  • 1
  • 10
12
votes
2 answers

Why is asyncio.Future incompatible with concurrent.futures.Future?

The two classes represent excellent abstractions for concurrent programming, so it's a bit disconcerting that they don't support the same API. Specifically, according to the docs: asyncio.Future is almost compatible with…
max
  • 40,904
  • 40
  • 170
  • 328
12
votes
1 answer

Using `concurrent.futures.Future` as promise

In the Python docs I see: concurrent.futures.Future... ...should not be created directly except for testing. And I want to use it as a promise in my code and I'm very surprised that it is not recommended to use it like this. My use case: I have…
Gill Bates
  • 9,135
  • 20
  • 59
  • 128
11
votes
3 answers

Can concurrent.futures.Future be converted to asyncio.Future?

I'm practicing asyncio after writing multithreaded code many years. Noticed something which i find it strange. Both in asyncio and in concurrent there is a Future object. from asyncio import Future from concurrent.futures import Future Guess each…
Aaron_ab
  • 2,715
  • 3
  • 20
  • 36
11
votes
3 answers

Exception handling in concurrent.futures.Executor.map

From https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator. The following snippet only…
user1008636
  • 2,109
  • 7
  • 21
  • 38
11
votes
2 answers

Detect failed tasks in concurrent.futures

I've been using concurrent.futures as it has a simple interface and let user easily control the max number of threads/processes. However, it seems like concurrent.futures hides failed tasks and continue the main thread after all tasks…
Erika L
  • 177
  • 3
  • 9
11
votes
2 answers

asyncio yield from concurrent.futures.Future of an Executor

I have a long_task function which runs a heavy cpu-bound calculation and I want to make it asynchronous by using the new asyncio framework. The resulting long_task_async function uses a ProcessPoolExecutor to offload work to a different process to…
krdx
  • 1,305
  • 14
  • 21
10
votes
1 answer

ProcessPoolExecutor logging fails to log inside function on Windows but not on Unix / Mac

When I run the following script on a Windows computer, I do not see any of the log messages from the log_pid function, however I do when I run on Unix / Mac. I've read before that multiprocessing is different on Windows compared to Mac, but it's not…
blahblahblah
  • 1,944
  • 5
  • 33
  • 54
10
votes
1 answer

How to chain futures in a non-blocking manner? That is, how to use one future as an input in another future without blocking?

Using the below example, how can future2 use the result of future1 once future1 is complete (without blocking future3 from being submitted)? from concurrent.futures import ProcessPoolExecutor import time def wait(seconds): time.sleep(seconds) …
Greg
  • 6,043
  • 8
  • 41
  • 93
10
votes
1 answer

Is it possible to pass Python Future objects between processes?

Based on my experiments I'm guessing the answer to this is no. But perhaps it might be possible with some changes to the futures module. I would like to submit a worker that itself creates an executor and submits work. I want to return that second…
Erotemic
  • 3,688
  • 4
  • 28
  • 59
10
votes
2 answers

Python parallel execution with selenium

I'm confused about parallel execution in python using selenium. There seems to be a few ways to go about it, but some seem out of date. I'm wondering what is the latest way to do parallel execution using selenium? There's a python module called…
Ke.
  • 2,224
  • 5
  • 32
  • 70
10
votes
2 answers

From concurrent.futures to asyncio

I have two problems with concurrent.futures: How to break time.sleep() in a python concurrent.futures? Conclusion: time.sleep() cannot be interrupted. One solution is: You can write a loop around it and do short sleeps. See How to break time.sleep()…
guettli
  • 26,461
  • 53
  • 224
  • 476
9
votes
2 answers

Python - Properly Kill/Exit Futures Thread?

I was previously using the threading.Thread module. Now I'm using concurrent.futures -> ThreadPoolExecutor. Previously, I was using the following code to exit/kill/finish a thread: def terminate_thread(thread): """Terminates a python thread from…
xendi
  • 1,831
  • 3
  • 27
  • 51
9
votes
4 answers

In Futures.transform, what is the difference between using a Function and an AsyncFunction

I know that the apply method of Function returns an object synchronously, and the apply of AsyncFunction runs asynchronously and returns a Future. Can you give me an example of when to prefer what. One code snippet that I saw looked something like…
SherinThomas
  • 1,691
  • 4
  • 15
  • 19
9
votes
1 answer

Why I cannot use python module concurrent.futures in class method?

I want to make my class method runs parallel, but it only produces some kind of error that I can not solve. My code is: import concurrent.futures as futures samples = ['asfd', 'zxcv', 'asf', 'qwer'] class test: def __init__(self, samples): …
user2028191
  • 91
  • 1
  • 2
1 2
3
…
46 47