python - Scrapy尝试抓取网页的信息内部链接

我正在尝试抓取页面JobStreet

我能够抓取主页上的信息，但当我尝试抓取页面的内部链接时，我的问题出现了，例如第一个帖子 here

这是我的代码片段:

import scrapy

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy import Item, Field

class IT(scrapy.Spider):
    name = 'IT'

    allowed_domains = ["www.jobstreet.com.sg"]
    start_urls = [
        'https://www.jobstreet.com.sg/en/job-search/job-vacancy.php?key=&specialization=191%2C192%2C193&area=&salary=&ojs=3&src=12',
    ]

    rules = (
        Rule(SgmlLinkExtractor(allow=[r'/en/job/*.'], restrict_xpaths=('//*[(@class = "position-title-link")]',)), callback='parse_info', follow=True)
    )


    def parse_info(self, response):

        self.logger.info('response.url=%s' % response.url)

无法从 parse_info 获得任何类型的响应。

最佳答案

你可以改变