0

I currently have an instance of scrapyd up and running locally on my machine. This instance of scrapyd needs to be available to other PC's on my employers network. I've read about scrapy-cloud (https://doc.scrapinghub.com/scrapy-cloud.html) and other cloud based services. However I'd much rather host scrapyd on our network, since the spiders I've built pull data from csv files stored on our servers.

I've searched through the scrapyd documentation (https://scrapyd.readthedocs.io/en/stable/) and understand how to install and run scrapyd. I am also comfortable with uploading scrapy projects to scrapyd and running specific spiders.

What steps do I need to take in order to make my scrapyd instance available to other machines on our network? All of our PC's and servers run on a windows OS

The answer doesn't need to be a specific step by step guide. I'm just looking for someone to point me in the right direction, because I am unsure how to proceed.

Eitan Seri-Levi
  • 316
  • 3
  • 12

1 Answers1

1

if you are in a lan in the same range of ip.

you can follow the manual and check your ip ifconfig in linux ipconfig in windows

and run the commands in the manual curl http://localhost:6800/addversion.json -F project=myproject -F version=r23 -F egg=@myproject.egg

and change the localhost with your ipaddress

for example if your ip is 192.168.1.10

you will run in other pc }.

curl http://192.168.1.10:6800/addversion.json -F project=myproject -F version=r23 -F egg=@myproject.egg

You need open the port if you use firewalls, and if you don't use cURL in windows can download and install it: How do I install/set up and use cURL on Windows?

More information about the api check the manual

Community
  • 1
  • 1
Jhovanny Uribe
  • 198
  • 9