Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

696 questions
70
votes
4 answers

How does ThreadPoolExecutor().map differ from ThreadPoolExecutor().submit?

I was just very confused by some code that I wrote. I was surprised to discover that: with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor: results = list(executor.map(f, iterable)) and with…
Patrick Collins
  • 9,307
  • 4
  • 23
  • 60
51
votes
2 answers

How do you kill Futures once they have started?

I am using the new concurrent.futures module (which also has a Python 2 backport) to do some simple multithreaded I/O. I am having trouble understanding how to cleanly kill tasks started using this module. Check out the following Python 2/3 script,…
Nick Chammas
  • 9,813
  • 7
  • 49
  • 105
47
votes
3 answers

Use tqdm with concurrent.futures?

I have a multithreaded function that I would like a status bar for using tqdm. Is there an easy way to show a status bar with ThreadPoolExecutor? It is the parallelization part that is confusing me. import concurrent.futures def f(x): return…
876868587
  • 2,802
  • 2
  • 16
  • 43
45
votes
1 answer

ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool

I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I've noticed that, almost with identical code, using the Pool from concurrent.futures is way slower than using multiprocessing.Pool. This is the version…
astrojuanlu
  • 5,095
  • 8
  • 36
  • 84
44
votes
2 answers

What is the difference between concurrent.futures and asyncio.futures?

To clarify the reason for this question: It is confusing to use two modules with the same name. What do they represent that makes them distinct? What task(s) can one solve that the other can't and vice-versa?
sargas
  • 4,422
  • 5
  • 43
  • 62
28
votes
1 answer

Number of max_workers when using ThreadPoolExecutor from concurrent.futures?

What are the factors to consider when deciding what to set max_workers to in ThreadPoolExecutor from concurrent.futures? As long as you can expect Python 3.5+ to be available, is there any reason not to set max_workers to None which will then…
edjohnsonwilliams
  • 381
  • 1
  • 3
  • 4
23
votes
2 answers

The workers in ThreadPoolExecutor is not really daemon

The thing I cannot figure out is that although ThreadPoolExecutor uses daemon workers, they will still run even if main thread exit. I can provide a minimal example in python3.6.4: import concurrent.futures import time def fn(): while True: …
Sraw
  • 14,837
  • 5
  • 37
  • 62
23
votes
2 answers

python concurrent.futures.ProcessPoolExecutor: Performance of .submit() vs .map()

I am using concurrent.futures.ProcessPoolExecutor to find the occurrence of a number from a number range. The intent is to investigate the amount of speed-up performance gained from concurrency. To benchmark performance, I have a control - a serial…
Sun Bear
  • 5,117
  • 5
  • 30
  • 71
23
votes
3 answers

Getting original line number for exception in concurrent.futures

Example of using concurrent.futures (backport for 2.7): import concurrent.futures # line 01 def f(x): # line 02 return x * x # line 03 data = [1, 2, 3, None, 5] # line 04 with concurrent.futures.ThreadPoolExecutor(len(data)) as executor: #…
djeendo
  • 287
  • 1
  • 2
  • 9
22
votes
2 answers

Finding the cause of a BrokenProcessPool in python's concurrent.futures

In a nutshell I get a BrokenProcessPool exception when parallelizing my code with concurrent.futures. No further error is displayed. I want to find the cause of the error and ask for ideas of how to do that. Full problem I am using…
Samufi
  • 1,801
  • 3
  • 12
  • 32
21
votes
1 answer

Nesting concurrent.futures.ThreadPoolExecutor

I have a program where I am currently using a concurrent.futures.ThreadPoolExecutor to run multiple tasks concurrently. These tasks are typically I/O bound, involving access to local databases and remote REST APIs. However, these tasks could…
Andrew McLean
  • 303
  • 3
  • 10
20
votes
3 answers

Python: concurrent.futures How to make it cancelable?

Python concurrent.futures and ProcessPoolExecutor provide a neat interface to schedule and monitor tasks. Futures even provide a .cancel() method: cancel(): Attempt to cancel the call. If the call is currently being executed and cannot be…
Ketzu
  • 583
  • 1
  • 5
  • 12
20
votes
2 answers

Checking up on a `concurrent.futures.ThreadPoolExecutor`

I've got a live concurrent.futures.ThreadPoolExecutor. I want to check its status. I want to know how many threads there are, how many are handling tasks and which tasks, how many are free, and which tasks are in the queue. How can I find out these…
Ram Rachum
  • 71,442
  • 73
  • 210
  • 338
19
votes
2 answers

What's the difference between python's multiprocessing and concurrent.futures?

A simple way of implementing multiprocessing in python is from multiprocessing import Pool def calculate(number): return number if __name__ == '__main__': pool = Pool() result = pool.map(calculate, range(4)) An alternative…
David Zwicker
  • 20,698
  • 6
  • 54
  • 72
19
votes
2 answers

`DummyExecutor` for Python's `futures`

Python's futures package allows us to enjoy ThreadPoolExecutor and ProcessPoolExecutor for doing tasks in parallel. However, for debugging it is sometimes useful to temporarily replace the true parallelism with a dummy one, which carries out the…
Ram Rachum
  • 71,442
  • 73
  • 210
  • 338
1
2 3
…
46 47