Python, Passing data in Scrapy

Question

How do I actually pass data into parse for my spider, le's say variable name or temp.

class CSpider(scrapy.Spider):

    name = "s1"
    allowed_domains = ["abc.com"]
    temp = ""
    start_urls = [
        url.strip() for url in lists
    ]
    def parse(self, response):
        //How do i pass data into here, eg name, temp

What kind of data you want to pass to the `parse` method and why? — damgad, Apr 14 '15 at 15:24
@damgad because the response doesnt contain what the data is about. Am i doing it wrong ? :o — CodeGuru, Apr 14 '15 at 21:39

score 1 · Accepted Answer · edited May 23 '17 at 10:26

1

If you are defining the temp variable as a class-level variable, you can access it via self.temp.

If this is something you want to be passed from a command-line, see the following topics:

edited May 23 '17 at 10:26

Community

1
1

answered Apr 14 '15 at 19:58

alecxe

414,977
106
935
1,083

score 0 · Answer 2 · answered Jan 02 '21 at 17:20

As alecxe answered, you can use attributes (class-level variables) to make variables or constants accessible wherever in your class or you can also add a parameter to your method (functions of a class) parse if you want to be able to give values to that parameter that would come from outside of the class.

I'll try here to give you an example of your code with both solutions.

Using an attribute:

class CSpider(scrapy.Spider):

    name = "s1"
    allowed_domains = ["abc.com"]
    temp = ""

    # Here is our attribute
    self.number_of_days_in_a_week = 7

    start_urls = [
        url.strip() for url in lists
    ]
    def parse(self, response):
        # It is now used in the method
        print(f"In a week, there is {self.number_of_days_in_a_week} days.")

If you need to, here is how to pass it as an other argument:

class CSpider(scrapy.Spider):

    name = "s1"
    allowed_domains = ["abc.com"]
    temp = ""
    start_urls = [
        url.strip() for url in lists
    ]
    def parse(self, what_you_want_to_pass_in):
        print(f"In a week, there is {what_you_want_to_pass_in} days.")

# We create an instance of the spider
spider1 = CSpider

# Then we use it's method with an argument
spider1.parse(7)

Note that in the second example, I took back the response argument from your parse method because it was easier to show how the arguments would be passed. Still, if you consider the entire Scrapy framework, you can for sure add external values using this solution.

Python, Passing data in Scrapy

2 Answers2