python - 将额外的参数传递给 scrapy.Request()

Question

实际上我想将与特定网站相关的所有数据(文本、hrefs、图像)存储到一个文件夹中。为了做到这一点，我需要将该文件夹的路径传递给所有不同的解析函数。所以我想像这样在 scrapy.Request() 中将此路径作为额外的 kwargs 传递:

yield scrapy.Request(url=url,dont_filter=True, callback=self.parse,errback = self.errback_function,kwargs={'path': '/path/to_folder'})

但它给出错误 TypeError: __init__() got an unexpected keyword argument 'kwargs'

如何将该路径传递给下一个函数？

Answer 1

对于任何可能需要它的人......

您可以像这样使用 meta 参数来传递额外的参数...

   yield scrapy.Request(url=url,dont_filter=True, 
callback=self.parse,errback = self.errback_function,  meta={'filepath': filepath})

更新:

Request.cb_kwargs was introduced in version 1.7. Prior to that, using Request.meta was recommended for passing information around callbacks. After 1.7, Request.cb_kwargs became the preferred way for handling user information, leaving Request.meta for communication with components like middlewares and extensions.

因此对于 >= 1.7 的版本，以下是可行的:

   request = scrapy.Request('http://www.example.com/index.html',
                             callback=self.parse_page2,
                             cb_kwargs=dict(main_url=response.url))

你可以引用这个文档:https://doc.scrapy.org/en/latest/topics/request-response.html#passing-additional-data-to-callback-functions