How do I pass form data with Scrapy from the command line?

Question

How could I pass username and password from the command line? Thanks!

class LoginSpider(Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'username': 'john', 'password': 'secret'},
                    callback=self.after_login)]

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.log("Login failed", level=log.ERROR)
            return

        # continue scraping with authenticated session...

score 7 · Answer 1 · answered Jun 25 '15 at 17:03

7

Open terminal and make sure scrapy installed.

scrapy shell
from scrapy.http import FormRequest
request=FormRequest(url='http://www.example.com/users/login.php',formdata={'username': 'john','password':'secret',})

Information:

Scrapy 1.0.0

answered Jun 25 '15 at 17:03

Joni

2,949
2
12
10

score 6 · Answer 2 · answered Jan 13 '14 at 21:41

you can do

scrapy crawl spidername -a username="john" -a password="secret"

and then

class LoginSpider(Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'username': self.username, 'password': self.password},
                    callback=self.after_login)]

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.log("Login failed", level=log.ERROR)
            return

        # continue scraping with authenticated session...

How do I pass form data with Scrapy from the command line?

2 Answers2