python - Access response from spider in items pipeline in scrapy -


i have spider

class productsspider(scrapy.spider):     name = "products"     allowed_domains = ["example.com"]     start_urls = [         'http://example.com/url'     ]      def parse(self, response): 

and have pipeline class

class productsdatapipeline(object):     """ item pipeline products data crawler """      def process_item(self, item, spider):            return item 

but want response argument of parse function in parse_item function without setting attribute item object,is possible

no it's not possible.

responses not forwarded pipelines. either have store response in item or use external storage store response , fetch in pipeline. second option better, , avoids many problems can result storing response in item (e.g. memory problems). example save response form of storage in parse callback, save reference storage in item field, , fetch response storage in pipeline.

but depends on trying do, response available in spider middleware process_spider_output perhaps can use instead of processing item in pipeline.


Comments

Popular posts from this blog

How has firefox/gecko HTML+CSS rendering changed in version 38? -

javascript - Complex json ng-repeat -

jquery - Cloning of rows and columns from the old table into the new with colSpan and rowSpan -