python - 无法弄清楚这只蜘蛛出了什么问题

标签 python scrapy

所以我努力确定我在这里做错了什么..希望抓取这个网站并提取每个项目的一些详细信息 - 类别 - 标题 - 价格 - url。但不明白为什么蜘蛛不会给我之后的结果。

如果我删除这个蜘蛛中的parse_details函数，蜘蛛将抓取我想要的url，但是当我调用parse_details来获取一些数据时，它会破坏抓取并且无法获取我正在寻找的完整范围。 (新手所以请温柔一点!)

没有我的 yield 请求

# -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url))



    def parse_details(self, response):

        pass

日志

2020-02-16 14:52:21 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:52:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 113681,
 'downloader/request_count': 178,
 'downloader/request_method_count/GET': 178,
 'downloader/response_bytes': 9932059,
 'downloader/response_count': 178,
 'downloader/response_status_count/200': 178,
 'dupefilter/filtered': 212,
 'elapsed_time_seconds': 17.984344,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 52, 21, 844590),
 'log_count/DEBUG': 179,
 'log_count/INFO': 10,
 'memusage/max': 53600256,
 'memusage/startup': 53600256,
 'request_depth_max': 4,
 'response_received_count': 178,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 177,
 'scheduler/dequeued/memory': 177,
 'scheduler/enqueued': 177,
 'scheduler/enqueued/memory': 177,
 'start_time': datetime.datetime(2020, 2, 16, 14, 52, 3, 860246)}
2020-02-16 14:52:21 [scrapy.core.engine] INFO: Spider closed (finished)

有 yield 请求

# -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)  

    def parse_details(self, response):
        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category,
                'title': title,
                'price': price,
                'url': url
            }

日志

2020-02-16 14:55:14 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:55:14 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 10801,
 'downloader/request_count': 19,
 'downloader/request_method_count/GET': 19,
 'downloader/response_bytes': 902364,
 'downloader/response_count': 19,
 'downloader/response_status_count/200': 19,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 2.775066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 55, 14, 607396),
 'item_scraped_count': 2,
 'log_count/DEBUG': 22,
 'log_count/INFO': 10,
 'memusage/max': 53420032,
 'memusage/startup': 53420032,
 'request_depth_max': 1,
 'response_received_count': 19,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 14, 55, 11, 832330)}
2020-02-16 14:55:14 [scrapy.core.engine] INFO: Spider closed (finished)

最佳答案

也许它不能解决你的问题，但把它放在这里。

我将所有内容放入单个文件 script.py 作为独立脚本，并使用 CrawlerProcess() 运行 

python script.py

我得到了包含一些数据的文件output.csv。但它只能在两页上找到项目。其他页面只有子类别列表。

脚本.py

from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)  

    def parse_details(self, response):
        # you can get it once - and use `strip()` to remove spaces
        category = response.xpath('//h1/text()').extract_first().strip()

        # you should search cards which keeps items separatelly - and don't extract it
        items = response.xpath('//div[@class="product-card__content"]') #.extract()
        print(len(items))

        for item in items:
            # you have to use `item.xpath` instead of `response.xpath`
            title = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = item.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            if title and price and url:
                yield {
                    'category': category,
                    'title': title.strip(),
                    'price': price.strip(),
                    'url': url.strip()
                }

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    #'USER_AGENT': 'Mozilla/5.0',
    'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0',
    # save in file CSV, JSON or XML
    'FEED_FORMAT': 'csv',     # csv, json, xml
    'FEED_URI': 'output.csv', #
})
c.crawl(WickesSpider)
c.start()

登录屏幕

2020-02-16 17:16:35 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: scrapybot)
2020-02-16 17:16:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.6 (default, Dec 19 2019, 23:50:13) - [GCC 7.4.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-5.3.0-28-generic-x86_64-with-LinuxMint-19.3-tricia
2020-02-16 17:16:35 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'csv', 'FEED_URI': 'output.csv', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0'}
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet Password: 7298b359c6ef9bcb
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-02-16 17:16:35 [scrapy.core.engine] INFO: Spider opened
2020-02-16 17:16:35 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-16 17:16:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/c/1000173> (referer: None)
2020-02-16 17:16:36 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2020-02-16 17:16:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Protective-Sheeting/c/1018000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
9
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm', 'price': '£45.00', 'url': '/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Fascias+Soffits/c/1000183> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m', 'price': '£1.50', 'url': '/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m', 'price': '£14.00', 'url': '/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m', 'price': '£2.00', 'url': '/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m', 'price': '£21.00', 'url': '/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056'}
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Bricks%2C-Blocks+Lintels/c/1000177> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Hardware-and-Metalwork/c/1000179> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Nails%2C-Screws+Adhesives/Sealants%2C-Adhesives+Fillers/c/1068000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Water+Damp-Proofing/c/1000185> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
34
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - White 10L', 'price': '£37.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - White 10L', 'price': '£45.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L', 'price': '£15.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Cappuccino 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L', 'price': '£30.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Ivory 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Classic Grey 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L', 'price': '£17.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Matt Emulsion - White 10L', 'price': '£12.00', 'url': '/Wickes-Matt-Emulsion---White-10L/p/145119'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L', 'price': '£20.00', 'url': '/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Magnolia 10L', 'price': '£23.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Magnolia 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394'}
0
0
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Brilliant White 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Shadow Grey 10L', 'price': '£25.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Undercoat Paint - White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 1L', 'price': '£18.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - White 2.5L', 'price': '£20.00', 'url': '/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L', 'price': '£38.00', 'url': '/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - Magnolia 10L', 'price': '£42.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Magnolia 15L', 'price': '£29.99', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L', 'price': '£32.00', 'url': '/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L', 'price': '£18.00', 'url': '/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L', 'price': '£40.00', 'url': '/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L', 'price': '£22.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L', 'price': '£60.00', 'url': '/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Mouldings/c/1000563> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cladding/c/1000182> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Roofing/c/1000181> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Insulation/c/1000184> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Guttering+Drainage/c/1000180> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cement+Aggregates/c/1000176> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Plaster+Plasterboard/c/1000178> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
0
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Sheet-Materials/c/1000174> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 17:16:39 [scrapy.extensions.feedexport] INFO: Stored csv feed (35 items) in: output.csv
2020-02-16 17:16:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 11242,
 'downloader/request_count': 18,
 'downloader/request_method_count/GET': 18,
 'downloader/response_bytes': 901446,
 'downloader/response_count': 18,
 'downloader/response_status_count/200': 18,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 3.808066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 16, 16, 39, 745213),
 'item_scraped_count': 35,
 'log_count/DEBUG': 54,
 'log_count/INFO': 11,
 'memusage/max': 50900992,
 'memusage/startup': 50900992,
 'request_depth_max': 1,
 'response_received_count': 18,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 16, 16, 35, 937147)}
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Spider closed (finished)

输出.csv

category,title,price,url
Building Materials Clearance,Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm,£45.00,/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798
Building Materials Clearance,Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m,£1.50,/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061
Building Materials Clearance,Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m,£14.00,/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057
Building Materials Clearance,Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m,£2.00,/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060
Building Materials Clearance,Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m,£21.00,/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056
Trade Paint,Dulux Trade Supermatt Emulsion Paint - White 10L,£37.00,/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - White 10L,£45.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472
Trade Paint,Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L,£15.00,/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480
Trade Paint,Wickes Contract Silk Emulsion Paint - Cappuccino 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L,£30.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082
Trade Paint,Wickes Contract Matt Emulsion Paint - Ivory 10L,£25.00,/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725
Trade Paint,Wickes Contract Silk Emulsion Paint - Classic Grey 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117
Trade Paint,Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L,£17.00,/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737
Trade Paint,Wickes Matt Emulsion - White 10L,£12.00,/Wickes-Matt-Emulsion---White-10L/p/145119
Trade Paint,Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L,£20.00,/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443
Trade Paint,Wickes Trade Contract Silk Emulsion Magnolia 10L,£23.00,/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743
Trade Paint,Wickes Basic Matt Emulsion Paint - Magnolia 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394
Trade Paint,Wickes Basic Matt Emulsion Paint - Brilliant White 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070
Trade Paint,Wickes Trade Contract Matt Emulsion Shadow Grey 10L,£25.00,/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722
Trade Paint,Dulux Trade Undercoat Paint - White 2.5L,£25.00,/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490
Trade Paint,Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L,£25.00,/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 1L,£18.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079
Trade Paint,Dulux Trade High Gloss Paint - White 2.5L,£20.00,/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496
Trade Paint,Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L,£38.00,/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170
Trade Paint,Dulux Trade Supermatt Emulsion Paint - Magnolia 10L,£42.00,/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481
Trade Paint,Wickes Contract Silk Emulsion Paint - Magnolia 15L,£29.99,/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744
Trade Paint,Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L,£32.00,/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725
Trade Paint,Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L,£18.00,/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789
Trade Paint,Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L,£40.00,/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498
Trade Paint,Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L,£22.00,/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733
Trade Paint,Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L,£60.00,/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487
<小时/>

编辑:您还必须在 parse_details 中搜索新链接

    def parse(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)  

    def parse_details(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)  

        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category.strip(),
                'title': title.strip(),
                'price': price.strip(),
                'url': url.strip()
            }

关于python - 无法弄清楚这只蜘蛛出了什么问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/60249794/

上一篇：python - 如何测试多个变量与单个值的相等性？

下一篇：python - 使用 API key 从 Flask RESTful API 获取

相关文章：

python - 提取列表中名称的前两个字符并连接成一个字符串？

web-scraping - Scrapy:无法通过登录抓取页面

python - 使用保存在本地系统中的 html 抓取文件

python - 使用 Webdriver 和 PhantomJS 记录 HTTP 流量

python - 如何将 RGB PIL 图像转换为具有 3 个 channel 的 numpy 数组？

python - 用系列更新数据框

html - 使用 scrapy 和 css 从 HTML 中提取特定值

Python:Fuzzywuzzy 不适用于外来字符

python - Scrapy - 如何管理 cookie/ session

python - 如何使用Scrapy获取域名？

©2024 IT工具网  联系我们