我一直在尝试通过构建 scrapers 来磨练我的 python 技能,最近从 bs4 切换到 scrapy,以便我可以使用它的多线程和下载延迟功能。我已经能够制作一个基本的抓取器并将数据输出到 csv,但是当我尝试添加递归功能时,我遇到了问题。我尝试遵循 Scrapy Recursive download of Content 的建议但不断收到以下错误:
调试:重试 http://medford.craigslist.org%20%5Bu'/cto/4359874426.html'%5D> DNS 查找失败:找不到地址
这让我觉得我尝试加入链接的方式不起作用,因为它将字符插入到网址中,但我不知道如何修复它。有什么建议吗?
代码如下:
#-------------------------------------------------------------------------------
# Name: module1
# Purpose:
#
# Author: CD
#
# Created: 02/03/2014
# Copyright: (c) CD 2014
# Licence: <your licence>
#-------------------------------------------------------------------------------
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.http import Request
from scrapy.selector import *
class PageSpider(BaseSpider):
name = "cto"
start_urls = ["http://medford.craigslist.org/cto/"]
rules = (Rule(SgmlLinkExtractor(allow=("index\d00\.html", ), restrict_xpaths=('//p[@class="nextpage"]' ,))
, callback="parse", follow=True), )
def parse(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.select("//span[@class='pl']")
for titles in titles:
item = CraigslistSampleItem()
item['title'] = titles.select("a/text()").extract()
item['link'] = titles.select("a/@href").extract()
url = "http://medford.craiglist.org %s" % item['link']
yield Request(url=url, meta={'item': item}, callback=self.parse_item_page)
def parse_item_page(self, response):
hxs = HtmlXPathSelector(response)
item = response.meta['item']
item['description'] = hxs.select('//section[@id="postingbody"]/text()').extract()
return item
最佳答案
结果是你的代码:
url = "http://medford.craiglist.org %s" % item['link']
生成:
http://medford.craigslist.org [u'/cto/4359874426.html']
item['link']
返回代码中的列表,而不是您期望的字符串。您需要这样做:
url = 'http://medford.craiglist.org{}'.format(''.join(item['link']))
关于python - 使用 Scrapy 在 Craigslist 上进行递归抓取,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/22182312/