python - Access response from spider in items pipeline in scrapy -
i have spider
class productsspider(scrapy.spider): name = "products" allowed_domains = ["example.com"] start_urls = [ 'http://example.com/url' ] def parse(self, response):
and have pipeline class
class productsdatapipeline(object): """ item pipeline products data crawler """ def process_item(self, item, spider): return item
but want response argument of parse function in parse_item function without setting attribute item object,is possible
no it's not possible.
responses not forwarded pipelines. either have store response in item or use external storage store response , fetch in pipeline. second option better, , avoids many problems can result storing response in item (e.g. memory problems). example save response form of storage in parse callback, save reference storage in item field, , fetch response storage in pipeline.
but depends on trying do, response available in spider middleware process_spider_output perhaps can use instead of processing item in pipeline.
Comments
Post a Comment