python - scrapy请求类的使用方法

Question

有人可以帮助我使用 scrapy 请求类发出请求吗

我已经试过了，但它不起作用:

from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
url = 'http://www.fetise.com'
a = Request(url)
hxs = HtmlXPathSelector(a)

错误是:

Traceback (most recent call last):
 File "sa.py", line 83, in <module>
 hxs = HtmlXPathSelector(a)
 File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmlsel.py",line 31,in __init__
_root = LxmlDocument(response, self._parser)
 File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 27,in __new__
cache[parser] = _factory(response, parser)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 13, in _factory
 body = response.body_as_unicode().strip().encode('utf8') or '<html/>'AttributeError: 'Request' object has no attribute 'body_as_unicode'` 

我知道回调......实际上我首先想从网站上删除 URL，然后将它们用作起始 URL......

Answer 1

请试试这个:

import urllib
from scrapy.selector import HtmlXPathSelector
from pprint import pprint

url = 'http://www.fetise.com'
data = urllib.urlopen(url).read()
hxs = HtmlXPathSelector(text=data)

lista = hxs.select('//ul[@class="categoryMenu"]/li/ul/li/a/@href').extract()

acb = ["http://www.fetise.com/" + i if "http://www.fetise.com/" not in i else i for i in lista] + [u"http://www.fetise.com/sale"]

pprint(acb)

这是输出:

[u'http://www.fetise.com/apparel/shirts',
 u'http://www.fetise.com/apparel/tees',
 u'http://www.fetise.com/apparel/tops-and-tees',
 u'http://www.fetise.com/accessories/belts',
 u'http://www.fetise.com/accessories/cufflinks',
 u'http://www.fetise.com/accessories/jewellery',
 u'http://www.fetise.com/accessories/lighters',
 u'http://www.fetise.com/accessories/others',
 u'http://www.fetise.com/accessories/sunglasses',
 u'http://www.fetise.com/accessories/ties-cufflinks',
 u'http://www.fetise.com/accessories/wallets',
 u'http://www.fetise.com/accessories/watches',
 u'http://www.fetise.com/footwear/boots',
 u'http://www.fetise.com/footwear/casual',
 u'http://www.fetise.com/footwear/flats',
 u'http://www.fetise.com/footwear/heels',
 u'http://www.fetise.com/footwear/loafers',
 u'http://www.fetise.com/footwear/sandals',
 u'http://www.fetise.com/footwear/shoes',
 u'http://www.fetise.com/footwear/slippers',
 u'http://www.fetise.com/footwear/sports',
 u'http://www.fetise.com/innerwear/boxers',
 u'http://www.fetise.com/innerwear/briefs',
 u'http://www.fetise.com/personal-care/deos',
 u'http://www.fetise.com/personal-care/haircare',
 u'http://www.fetise.com/personal-care/perfumes',
 u'http://www.fetise.com/personal-care/personal-care',
 u'http://www.fetise.com/personal-care/shavers',
 u'http://www.fetise.com/apparel/tees/gifts-for-her',
 u'http://www.fetise.com/footwear/sandals/gifts-for-her',
 u'http://www.fetise.com/footwear/shoes/gifts-for-her',
 u'http://www.fetise.com/footwear/heels/gifts-for-her',
 u'http://www.fetise.com/footwear/flats/gifts-for-her',
 u'http://www.fetise.com/footwear/ballerinas/gifts-for-her',
 u'http://www.fetise.com/footwear/loafers/gifts-for-her',
 u'http://www.fetise.com/sale']