0

I have been working on threads and trying to find out why my code do not like to work concurrently.

import concurrent.futures
import time
from queue import Queue

import requests

q = Queue()

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        ]


def explore_link(queue: Queue, url: str):
    response = requests.get(url)
    if "http://www.cnn.com/" in url:
        time.sleep(5)

    queue.task_done()
    return len(response.text)


# Start up
def main():
    for url in URLS:
        q.put(url)

    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        # start a future for a thread which sends work in through the queue
        while not q.empty():
            url = q.get()
            future = executor.submit(explore_link, queue=q, url=url)
            print(url, future.result())


if __name__ == '__main__':
    main()

Currently it does get stuck at whenever it reaches the

if "http://www.cnn.com/" in url:
        time.sleep(5)

it stops and wait for it to be finished meaning that it does work as synchronized. What I want to do is that I would like to run concurrently meaning that when we do reach the if-statement, instead of waiting, the next in the queue should be executed.

My question is how can I run this script concurrently without being blocked from each other executions?

ProtractorNewbie
  • 405
  • 2
  • 13
  • Does this answer your question? https://stackoverflow.com/questions/56729764/python-3-7-asyncio-sleep-and-time-sleep – ForceBru May 14 '21 at 14:51
  • Hi! @ForceBru Unfortunately I wont be able to use async as I will in future projects be using a package that is based on sync :'( Meaning I wont be able to use async for that – ProtractorNewbie May 14 '21 at 14:57

0 Answers0