python - Scrapy 关注链接

Question

我有以下 CrawlSpider，但无法访问大学网站上的链接。我认为这是因为加价不稳定，但我不确定。我尝试添加一条规则，但它不会遵循。我怎样才能做到这一点？

它作为单页蜘蛛运行，可以正常抓取第 1 页，但不跟踪链接。

注意，不是作业，只是我玩玩并得到了抓取 Dmoz 的板。感谢所有帮助。

# -*- coding: utf-8 -*-
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

from example.items import ExampleItem

class ExampleSpider(CrawlSpider):
    name = "example"
    allowed_domains = ["example.ac.uk"]
    start_urls = (
        'http://www.example.ac.uk/courses/course-finder?query=&f.Year_of_entry|E=2015/16&f.Type|D=Undergraduate',
        ''
    )

    rules = (Rule (SgmlLinkExtractor(allow=("index\.php", ), callback="parse"),))

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//div[@id="course_list"]')
        items = []

        for site in sites:
            item = ExampleItem()
            item['link'] = site.xpath('//h2/a/@href').extract()
            item['name'] = site.xpath('//h2/a/text()').extract()
            items.append(item)

        return items

网站上的分页标记如下:

   <div class="pagination">
            <ul>
                <li><i class="fa fa-chevron-left"></i><span>Previous</span></li>

                    <li><span>Go to page</span> 1</li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=11"><span>Go to page</span> 2</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=21"><span>Go to page</span> 3</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=31"><span>Go to page</span> 4</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=41"><span>Go to page</span> 5</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=51"><span>Go to page</span> 6</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=61"><span>Go to page</span> 7</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=71"><span>Go to page</span> 8</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=81"><span>Go to page</span> 9</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=91"><span>Go to page</span> 10</a></li>


                    <li><a href="course-finder?query=&amp;fYear_of_entryE=2015/16&amp;fTypeD=Undergraduate&amp;start_rank=11"><i class="fa fa-chevron-right"></i><span>Next</span></a></li>
            </ul>
        </div>

Answer 1

至少您遇到的第一个问题是您在链接提取器内定义回调，但应该在规则级别上定义:

rules = (
    Rule(LinkExtractor(allow=("index\.php", )), callback="parse_result"),
)

def parse_result(self, response):
    ...

此外，您需要一个单独的规则来遵循分页:

rules = (
    Rule(LinkExtractor(allow=("index\.php", )), callback="parse_result"),
    Rule(LinkExtractor(restrict_xpaths='//div[@class="pagination"]'), follow=True),
)