web-scraping - Scrapy:每个蜘蛛一个 jobdir

Question

我有一个包含多个蜘蛛的 Scrapy 项目。如何使用自己的 jobdir 运行所有蜘蛛程序？我在下面的代码中展示了如何使用一个 jobdir 执行所有蜘蛛。

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

settings = get_project_settings()
settings.set('JOBDIR', 'saved_crawl', priority='cmdline')


process = CrawlerProcess(settings)

process.crawl('spider1')
process.crawl('spider2')
process.start()

Answer 1

好吧，我发现解决方案很简单。我只需要在自定义设置字典中为每个蜘蛛定义 JOBDIR :

class Spider1(scrapy.Spider):
    name = 'spider1'
    custom_settings = {'JOBDIR': 'crawl_spider1'}