I need to run the same spider as many times as the user requests it from script. it works fine if I run it only once, but after that I get twisted.internet.error.ReactorNotRestartable
. here is my code:
from scrapy.crawler import CrawlerRunner
from crawlers.spiders import MySpider
from scrapy.utils.project import get_project_settings
from twisted.internet import reactor
def run_spider(params):
runner = CrawlerRunner(get_project_settings())
@defer.inlineCallbacks
def crawl():
yield runner.crawl(MySpider, params)
reactor.stop()
crawl()
reactor.run()
if __name__ == "__main__":
run_spider(param_1)
if I run run_spider(param_2)
after the first one, i get the error.ReactorNotRestartable()
error. I've seen implementations using for loops with the spider inside the run_spider()
function, but for my case the important thing to remember is that I don't know how many times the user is going to run the spider since the request comes from an API.