我正在使用 scrapy 抓取我的站点地图,以检查 404、302 和 200 个页面。但我似乎无法获得响应代码。到目前为止,这是我的代码:
from scrapy.contrib.spiders import SitemapSpider
class TothegoSitemapHomesSpider(SitemapSpider):
name ='tothego_homes_spider'
## robe che ci servono per tothego ##
sitemap_urls = []
ok_log_file = '/opt/Workspace/myapp/crawler/valid_output/ok_homes'
bad_log_file = '/opt/Workspace/myapp/crawler/bad_homes'
fourohfour = '/opt/Workspace/myapp/crawler/404/404_homes'
def __init__(self, **kwargs):
SitemapSpider.__init__(self)
if len(kwargs) > 1:
if 'domain' in kwargs:
self.sitemap_urls = ['http://url_to_sitemap%s/sitemap.xml' % kwargs['domain']]
if 'country' in kwargs:
self.ok_log_file += "_%s.txt" % kwargs['country']
self.bad_log_file += "_%s.txt" % kwargs['country']
self.fourohfour += "_%s.txt" % kwargs['country']
else:
print "USAGE: scrapy [crawler_name] -a country=[country] -a domain=[domain] \nWith [crawler_name]:\n- tothego_homes_spider\n- tothego_cars_spider\n- tothego_jobs_spider\n"
exit(1)
def parse(self, response):
try:
if response.status == 404:
## 404 tracciate anche separatamente
self.append(self.bad_log_file, response.url)
self.append(self.fourohfour, response.url)
elif response.status == 200:
## printa su ok_log_file
self.append(self.ok_log_file, response.url)
else:
self.append(self.bad_log_file, response.url)
except Exception, e:
self.log('[eccezione] : %s' % e)
pass
def append(self, file, string):
file = open(file, 'a')
file.write(string+"\n")
file.close()
从scrapy的文档中,他们说 response.status 参数是一个整数,对应于响应的状态码。到目前为止,它只记录 200 个状态 url,而 302 没有写入输出文件(但我可以在 crawl.log 中看到重定向)。那么,我该怎么做才能“捕获” 302 请求并保存这些 url?
最佳答案
假设默认蜘蛛中间件已启用,200-300 范围之外的响应代码会被 HttpErrorMiddleware 过滤掉。您可以通过设置蜘蛛上的 handle_httpstatus_list 属性来告诉中间件您要处理 404。
class TothegoSitemapHomesSpider(SitemapSpider):
handle_httpstatus_list = [404]
关于python - Scrapy 和响应状态码 : how to check against it?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/9698372/