python - Scrapy:尝试从不正确的选择器列表中提取数据

Question

我正在尝试从网站上抓取足球赛程，但我的蜘蛛不太正确，因为我要么为所有选择器或 homeTeam 和 awayTeam 变量重复相同的赛程是巨大的数组，分别包含所有主场或客场。无论哪种方式，它都应该反射(reflect)主场 vs 客场的格式。

这是我目前的尝试:

class FixtureSpider(CrawlSpider):
    name = "fixturesSpider"
    allowed_domains = ["www.bbc.co.uk"]
    start_urls = [
        "http://www.bbc.co.uk/sport/football/premier-league/fixtures"
    ]

    def parse(self, response):
        for sel in response.xpath('//table[@class="table-stats"]/tbody/tr[@class="preview"]'):

        item = Fixture()
        item['kickoff'] =  str(sel.xpath("//table[@class='table-stats']/tbody/tr[@class='preview']/td[3]/text()").extract()[0].strip())
        item['homeTeam'] = str(sel.xpath("//table[@class='table-stats']/tbody/tr/td[2]/p/span/a/text()").extract()[0].strip())
        item['awayTeam'] = str(sel.xpath("//table[@class='table-stats']/tbody/tr/td[2]/p/span/a/text()").extract()[1].strip())
        yield item

这会重复返回以下不正确的信息:

2015-03-20 21:41:40+0000 [fixturesSpider] DEBUG: Scraped from <200 http://www.bbc.co.uk/sport/football/premier-league/fixtures>
{'awayTeam': 'West Brom', 'homeTeam': 'Man City', 'kickoff': '12:45'}
2015-03-20 21:41:40+0000 [fixturesSpider] DEBUG: Scraped from <200 http://www.bbc.co.uk/sport/football/premier-league/fixtures>
{'awayTeam': 'West Brom', 'homeTeam': 'Man City', 'kickoff': '12:45'}

有人可以告诉我哪里出错了吗？

Answer 1

问题是您在循环中使用的 XPath 表达式是绝对的 - 它们从根元素开始，但应该相对于 sel 指向的当前行。换句话说，您需要在当前行上下文中搜索。

固定版本:

for sel in response.xpath('//table[@class="table-stats"]/tbody/tr[@class="preview"]'):
    item = Fixture()
    item['kickoff'] =  str(sel.xpath("td[3]/text()").extract()[0].strip())
    item['homeTeam'] = str(sel.xpath("td[2]/p/span/a/text()").extract()[0].strip())
    item['awayTeam'] = str(sel.xpath("td[2]/p/span/a/text()").extract()[1].strip())
    yield item

这是我得到的输出:

{'awayTeam': 'West Brom', 'homeTeam': 'Man City', 'kickoff': '12:45'}
{'awayTeam': 'Swansea', 'homeTeam': 'Aston Villa', 'kickoff': '15:00'}
{'awayTeam': 'Arsenal', 'homeTeam': 'Newcastle', 'kickoff': '15:00'}
...

---

如果你想抓取比赛日期，你需要改变策略 - 迭代日期(h2 元素与 table-header 类)并获得第一个以下兄弟 table 元素:

for date in response.xpath('//h2[@class="table-header"]'):
    matches = date.xpath('.//following-sibling::table[@class="table-stats"][1]/tbody/tr[@class="preview"]')
    date = date.xpath('text()').extract()[0].strip()

    for match in matches:
        item = Fixture()
        item['date'] = date
        item['kickoff'] = match.xpath("td[3]/text()").extract()[0].strip()
        item['homeTeam'] = match.xpath("td[2]/p/span/a/text()").extract()[0].strip()
        item['awayTeam'] = match.xpath("td[2]/p/span/a/text()").extract()[1].strip()
        yield item