python-3.x - 我可以在不调用蜘蛛的情况下使用Scrapy吗？

Question

我已经安装了 scrapy 并按照网站上的 scrapy 教程进行操作。我注意到您需要创建一个蜘蛛，然后调用该蜘蛛才能使用它。我在想:我可以在普通的 python 脚本(在 IDLE 中)中使用 scrapy 命令吗？

Answer 1

@seppukki，最近我使用 CrawlerProcess 库尝试了类似的方法。我成功了。最小代表如下所示。可以查看笔记本here 。本质上，以下代码片段将抓取网页，提取内容并将其写入文件。我不必在终端中执行 scrapy 命令。

# import scrapy
import scrapy
from scrapy.crawler import CrawlerProcess

# Set up a pipeline
import json

class JsonWriterPipeline(object):

    def open_spider(self, spider):
        self.file = open('results\\quoteresult.txt', 'w+')

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item
# Define the spider
import logging

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]
    custom_settings = {
        'LOG_LEVEL': logging.WARNING,
        'ITEM_PIPELINES': {'__main__.JsonWriterPipeline': 1}, # Used for pipeline 1
        'FEED_FORMAT':'json',                                 # Used for pipeline 2
        'FEED_URI': 'quoteresult.json'                        # Used for pipeline 2
    }

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('span small::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }
# Start the crawl
process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(QuotesSpider)
process.start()

2018-09-22 18:07:39 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)
2018-09-22 18:07:39 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 2018), cryptography 2.2.2, Platform Windows-7-6.1.7601-SP1
2018-09-22 18:07:39 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'FEED_URI': 'quoteresult.json', 'LOG_LEVEL': 30, 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'}