我遇到一个问题,我的 scrapy 蜘蛛都不会抓取网站,只是抓取一页并抓取。我的印象是 rules
成员变量对此负责,但我无法让它跟踪任何链接。我一直在关注这里的文档:http://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider
我可能遗漏了什么,导致我的机器人无法爬行?
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import Selector
from Example.items import ExItem
class ExampleSpider(CrawlSpider):
name = "example"
allowed_domains = ["example.ac.uk"]
start_urls = (
'http://www.example.ac.uk',
)
rules = ( Rule (LinkExtractor(allow=("", ),),
callback="parse_items", follow= True),
)
最佳答案
将您的规则替换为以下规则:
rules = ( Rule(LinkExtractor(allow=('course-finder', ),restrict_xpaths=('//div[@class="pagination"]',)), callback='parse_items',follow=True), )
关于python - Scrapy 关注并抓取下一页,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/28805663/