1

I need to run the same spider as many times as the user requests it from script. it works fine if I run it only once, but after that I get twisted.internet.error.ReactorNotRestartable. here is my code:

from scrapy.crawler import CrawlerRunner
from crawlers.spiders import MySpider
from scrapy.utils.project import get_project_settings
from twisted.internet import reactor

def run_spider(params):

    runner = CrawlerRunner(get_project_settings())

    @defer.inlineCallbacks
    def crawl():
        yield runner.crawl(MySpider, params)
        reactor.stop()

    crawl()
    reactor.run()

if __name__ == "__main__":
    run_spider(param_1)

if I run run_spider(param_2) after the first one, i get the error.ReactorNotRestartable() error. I've seen implementations using for loops with the spider inside the run_spider() function, but for my case the important thing to remember is that I don't know how many times the user is going to run the spider since the request comes from an API.

Amir Imani
  • 1,927
  • 15
  • 20

0 Answers0