I have been working on threads and trying to find out why my code do not like to work concurrently.
import concurrent.futures
import time
from queue import Queue
import requests
q = Queue()
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
]
def explore_link(queue: Queue, url: str):
response = requests.get(url)
if "http://www.cnn.com/" in url:
time.sleep(5)
queue.task_done()
return len(response.text)
# Start up
def main():
for url in URLS:
q.put(url)
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
# start a future for a thread which sends work in through the queue
while not q.empty():
url = q.get()
future = executor.submit(explore_link, queue=q, url=url)
print(url, future.result())
if __name__ == '__main__':
main()
Currently it does get stuck at whenever it reaches the
if "http://www.cnn.com/" in url:
time.sleep(5)
it stops and wait for it to be finished meaning that it does work as synchronized. What I want to do is that I would like to run concurrently meaning that when we do reach the if-statement, instead of waiting, the next in the queue should be executed.
My question is how can I run this script concurrently without being blocked from each other executions?