我正在关注本指南 http://doc.scrapy.org/en/0.16/topics/practices.html#run-scrapy-from-a-script从我的脚本运行scrapy。 这是我的脚本的一部分:
crawler = Crawler(Settings(settings))
crawler.configure()
spider = crawler.spiders.create(spider_name)
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
print "It can't be printed out!"
它应该起作用:访问页面,抓取所需的信息并将输出 json 存储在我告诉它的位置(通过 FEED_URI)。但是当蜘蛛完成他的工作时(我可以通过输出 json 中的数字看到它)我的脚本的执行不会恢复。 可能不是scrapy问题。答案应该在扭曲的 react 堆的某个地方。 如何释放线程执行?
最佳答案
蜘蛛完成后,您需要停止 react 器。您可以通过监听 spider_closed
信号来完成此操作:
from twisted.internet import reactor
from scrapy import log, signals
from scrapy.crawler import Crawler
from scrapy.settings import Settings
from scrapy.xlib.pydispatch import dispatcher
from testspiders.spiders.followall import FollowAllSpider
def stop_reactor():
reactor.stop()
dispatcher.connect(stop_reactor, signal=signals.spider_closed)
spider = FollowAllSpider(domain='scrapinghub.com')
crawler = Crawler(Settings())
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
log.msg('Running reactor...')
reactor.run() # the script will block here until the spider is closed
log.msg('Reactor stopped.')
命令行日志输出可能类似于:
stav@maia:/srv/scrapy/testspiders$ ./api
2013-02-10 14:49:38-0600 [scrapy] INFO: Running reactor...
2013-02-10 14:49:47-0600 [followall] INFO: Closing spider (finished)
2013-02-10 14:49:47-0600 [followall] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 23934,...}
2013-02-10 14:49:47-0600 [followall] INFO: Spider closed (finished)
2013-02-10 14:49:47-0600 [scrapy] INFO: Reactor stopped.
stav@maia:/srv/scrapy/testspiders$
关于python - 从脚本抓取抓取总是在抓取后阻止脚本执行,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/14777910/