我写了一个基本的 Scrapy 蜘蛛来抓取一个似乎运行良好的网站,除了它不想停止的事实,即它不断重新访问相同的 url 并返回相同的内容 - 我总是最终得到阻止它。我怀疑它一遍又一遍地访问相同的网址。有没有规则可以阻止这种情况?或者还有什么我必须做的吗?也许是中间件?
蜘蛛如下:
class LsbuSpider(CrawlSpider):
name = "lsbu6"
allowed_domains = ["lsbu.ac.uk"]
start_urls = [
"http://www.lsbu.ac.uk"
]
rules = [
Rule(SgmlLinkExtractor(allow=['lsbu.ac.uk/business-and-partners/.+']), callback='parse_item', follow=True),
]
def parse_item(self, response):
join = Join()
sel = Selector(response)
bits = sel.xpath('//*')
scraped_bits = []
for bit in bits:
scraped_bit = LsbuItem()
scraped_bit['title'] = scraped_bit.xpath('//title/text()').extract()
scraped_bit['desc'] = join(bit.xpath('//*[@id="main_content_main_column"]//text()').extract()).strip()
scraped_bits.append(scraped_bit)
return scraped_bits
我的 settings.py
文件如下所示
BOT_NAME = 'lsbu6'
DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'
DUPEFILTER_DEBUG = True
SPIDER_MODULES = ['lsbu.spiders']
NEWSPIDER_MODULE = 'lsbu.spiders'
任何关于停止它连续运行的帮助/指导/说明将不胜感激......
因为我是新手;任何关于整理代码的评论也会有所帮助(或良好指导的链接)。
谢谢...
最佳答案
DupeFilter 默认启用:http://doc.scrapy.org/en/latest/topics/settings.html#dupefilter-class它基于请求 url。
我在没有任何自定义配置的新 Vanilla scrapy 项目上尝试了您的蜘蛛的简化版本。 dupefilter 起作用了,在几次请求后爬网停止了。我会说你的设置或你的 scrapy 版本有问题。我建议你升级到 scrapy 1.0,以确保:)
$ pip install scrapy --pre
我测试的简化蜘蛛:
from scrapy.spiders import CrawlSpider
from scrapy.linkextractors import LinkExtractor
from scrapy import Item, Field
from scrapy.spiders import Rule
class LsbuItem(Item):
title = Field()
url = Field()
class LsbuSpider(CrawlSpider):
name = "lsbu6"
allowed_domains = ["lsbu.ac.uk"]
start_urls = [
"http://www.lsbu.ac.uk"
]
rules = [
Rule(LinkExtractor(allow=['lsbu.ac.uk/business-and-partners/.+']), callback='parse_item', follow=True),
]
def parse_item(self, response):
scraped_bit = LsbuItem()
scraped_bit['url'] = response.url
yield scraped_bit
关于python - 停止 Scrapy 抓取相同的 URL,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29883132/