爬虫有一个最大的返回次数,记录在案here .到达那里后,我收到了类似于以下内容的错误:
Gave up retrying <GET https:/foo/bar/123> (failed 3 times)
我相信消息是由代码 here 产生的.
但是,我想对放弃做一些总结。具体来说,我想知道是否可以:
- 提取
123
URL 的一部分(ID)并将这些 ID 写到一个单独的文件中。 - 访问
meta
原文信息request
. This documentation可能会有帮助。
最佳答案
您可以子类化 scrapy.contrib.downloadermiddleware.retry.RetryMiddleware
并覆盖 _retry()
以对请求做任何您想做的事情而不是放弃。
from scrapy.contrib.downloadermiddleware.retry import RetryMiddleware
from scrapy import log
class CustomRetryMiddleware(RetryMiddleware):
def _retry(self, request, reason, spider):
retries = request.meta.get('retry_times', 0) + 1
if retries <= self.max_retry_times:
log.msg(format="Retrying %(request)s (failed %(retries)d times): %(reason)s",
level=log.DEBUG, spider=spider, request=request, retries=retries, reason=reason)
retryreq = request.copy()
retryreq.meta['retry_times'] = retries
retryreq.dont_filter = True
retryreq.priority = request.priority + self.priority_adjust
return retryreq
else:
# do something with the request: inspect request.meta, look at request.url...
log.msg(format="Gave up retrying %(request)s (failed %(retries)d times): %(reason)s",
level=log.DEBUG, spider=spider, request=request, retries=retries, reason=reason)
然后就是在您的 settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None,
'myproject.middlewares.CustomRetryMiddleware': 500,
}
关于python - Scrapy:收集重试消息,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/27793552/