我正在尝试抓取一个网站,html 的内容看起来像这样
<div class="panel-heading" role="tab" id="heading727654">
<h4 class="panel-title">
<a class="collapsed" data-toggle="collapse" data-parent="#accordion" href="#collapse727654" aria-expanded="false" aria-controls="collapse727654">
<div class="product-name">
<span class="product-title">
Aubrey<br><i>AGE DEFYING THERAPY CLEANSER 3.4 OZ</i>
</span>
</div>
<div class="product-price">
<span>
$10.99 / 3.40 OZ
</span>
</a>
</h4>
</div>
<div class="panel-heading" role="tab" id="heading727655">
<h4 class="panel-title">
<a class="collapsed" data-toggle="collapse" data-parent="#accordion" href="#collapse727655" aria-expanded="false" aria-controls="collapse727654">
<div class="product-name">
<span class="product-title">
Aubrey<br><i>AGE DEFYING THERAPY LIQUID</i>
</span>
</div>
<div class="product-price">
<span>
$12.99 / 4.40 OZ
</span>
</a>
</h4>
</div>
我的Python代码片段提取它是这样的
def parse(self, response):
filename = response.url.split("/")[-2] + '.html'
with open(filename, 'wb') as f:
for node in response.xpath('//div[re:test(@class, "panel-heading")]'):
print node.xpath('//span[re:test(@class, "product-title")]//text()').extract()
print node.xpath('//span[re:test(@class, "product-price")]//text()').extract()
当我在Python中运行上面的scrapy代码时,我没有得到预期的输出,相同的内容被重复了100次。有人可以帮我解决这个问题吗?
最佳答案
您需要在内部 XPath 表达式中添加点,以使它们在 node
上下文中工作。否则,搜索从树的根开始:
def parse(self, response):
filename = response.url.split("/")[-2] + '.html'
with open(filename, 'wb') as f:
for node in response.xpath('//div[re:test(@class, "panel-heading")]'):
print node.xpath('.//span[re:test(@class, "product-title")]//text()').extract()
print node.xpath('.//span[re:test(@class, "product-price")]//text()').extract()
关于python - 如何使用python迭代scrapy中的节点,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/37663181/