2

The function file_path parameter in imagepipiline responds. When I return str directly, I can successfully download the image. If response.meta.get('file_name') is used, the download will fail. It is also a string. Why can't the variable be used?

item.py -

class ImageItem(scrapy.Item):
    file_name = scrapy.Field()
    url = scrapy.Field()

test.py -

class TestSpider(scrapy.Spider):
    name = 'test'
    allowed_domains = ['www.baidu.com']
    start_urls = ['http://www.baidu.com/']
    def parse(self, response):
                image_item['file_name'] = 'yoxo.jpg'
        image_item['url'] = 'https://zmtimg.dfcfw.com/529d2d0f7a57c9135eb956b9afa1810a.em/art'
        yield image_item

pipelines.py -

class ArticleImagePipeline(ImagesPipeline):
    def file_path(self, request, response=None, info=None):
        try:
            print('response.file_name:{}'.format(response.meta.get('file_name')))
            print(type(response.meta.get('file_name')))
            print(response.meta.get('file_name'))
        except Exception as e:
            pass
        return 'hello.jpg'
    def get_media_requests(self, item, info):
        meta = {
            'file_name': item['file_name']
        }
        yield Request(url=item['url'], meta=meta)

when i'm running it .it successful!The results are as follows:

response.file_name:yoxo.jpg
<class 'str'>
yoxo.jpg
response.file_name:yoxo.jpg
<class 'str'>
yoxo.jpg

But when I use a variable as the return value, it failed.

def file_path(self, request, response=None, info=None):
    try:
        print('response.file_name:{}'.format(response.meta.get('file_name')))
        print(type(response.meta.get('file_name')))
        print(response.meta.get('file_name'))
    except Exception as e:
        pass
    return 'hello.jpg'

result:

2018-12-13 22:11:39 [scrapy.core.scraper] WARNING: Dropped: Image Downloaded Failed
{'file_name': 'yoxo.jpg',
 'url': 'https://zmtimg.dfcfw.com/529d2d0f7a57c9135eb956b9afa1810a.em/art'}

I tried it with the following code. The result is empty, why? The same is the first response normal return, written outside the try is empty

try:
    # aa = response.meta.get('file_name')
    print('response.file_name:{}'.format(response.meta.get('file_name')))
    print(type(response.meta.get('file_name')))
    print(response.meta.get('file_name'))
except Exception as e:
    logging.error(response)
    logging.error(e)
    pass
print(response.meta.get('file_name','yep'))
return 'hello.jpg'

result:

2018-12-13 22:15:31 [root] ERROR: None
2018-12-13 22:15:31 [root] ERROR: 'NoneType' object has no attribute 'meta'
2018-12-13 22:15:31 [scrapy.core.scraper] WARNING: Dropped: Image Downloaded Failed
{'file_name': 'yoxo.jpg',
 'url': 'https://zmtimg.dfcfw.com/529d2d0f7a57c9135eb956b9afa1810a.em/art'}

why?I couldn't understand 。 Many thanks for any help~

Tushar Walzade
  • 3,182
  • 4
  • 26
  • 45
S.DZ
  • 21
  • 1

0 Answers0