所以...在遵循这篇文章后，我正在我的 ubuntu ec2 实例上运行 scrapyd:http://www.dataisbeautiful.io/deploying-scrapy-ec2/
但是我想我无法让 pymongo 连接到我的 MongoLabs mongo 数据库，因为 ubuntu ec2 scrapyd 日志显示
pymongo.errors.ConnectionFailure: timed out
在后端方面，我是一个真正的菜鸟，所以我真的不知道是什么导致了这个问题。当我从本地主机运行 scrapyd 时，它工作得很好，并将抓取的数据保存到我的 MongoLabs 数据库中。对于在 ec2 实例上运行的 scrapyd，我可以通过在端口 6800 处输入 ec2 地址(相当于 scrapyd 的 localhost:6800)来访问 scrapyd gui，仅此而已。冰壶
curl http://aws-ec2-link:6800/schedule.json -d project=sportslab_scrape -d spider=max -d max_url="http://www.maxpreps.com/high-schools/de-la-salle-spartans-(concord,ca)/football/stats.htm"
返回“状态”:“好的”，我可以看到作业出现，但没有生成任何项目，日志仅显示
2014-11-17 02:20:13+0000 [scrapy] INFO: Scrapy 0.24.4 started (bot: sportslab_scrape_outer)
2014-11-17 02:20:13+0000 [scrapy] INFO: Optional features available: ssl, http11
2014-11-17 02:20:13+0000 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'sportslab_scrape.spiders', 'SPIDER_MODULES': ['sportslab_scrape.spiders'], 'FEED_URI': 'items/sportslab_scrape/max/4299afa26e0011e4a543060f585a893f.jl', 'LOG_FILE': 'logs/sportslab_scrape/max/4299afa26e0011e4a543060f585a893f.log', 'BOT_NAME': 'sportslab_scrape_outer'}
2014-11-17 02:20:13+0000 [scrapy] INFO: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-11-17 02:20:13+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-11-17 02:20:13+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
有人对我的问题有一些有用的见解吗？谢谢!
编辑:添加了连接代码。 设置.py
MONGODB_HOST = 'mongodb://user:<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="b9c9d8cacaf9d8cadddf97d4d6d7ded6d5d8db97dad6d4" rel="noreferrer noopener nofollow">[email protected]</a>:38839/sportslab_mongodb'
MONGODB_PORT = 38839 # Change in prod
MONGODB_DATABASE = "sportslab_mongodb" # Change in prod
MONGODB_COLLECTION = "sportslab"
Scrapy的Pipeline.py
from pymongo import Connection
from scrapy.conf import settings
class MongoDBPipeline(object):
def __init__(self):
connection = Connection(settings['MONGODB_HOST'], settings['MONGODB_PORT'])
db = connection[settings['MONGODB_DATABASE']]
self.collection = db[settings['MONGODB_COLLECTION']]
def process_item(self, item, spider):
self.collection.insert(dict(item))
return item
最佳答案
我解决了这个问题。最初，我将 ec2 安全组的出站规则设置为:
Outbound
Type:HTTP, Protocol: TCP, Port Range:80, Destination: 0.0.0.0/0
Type:Custom, Protocol: TCP, Port Range: 6800, Destination: 0.0.0.0/0
Type:HTTPS, Protocol: TCP, Port Range:443, Destination 0.0.0.0/0
但是，这还不够，因为我还需要一个特定的自定义 TCP 协议(protocol)来用于我连接到的 mongolab 数据库的实际端口，它应该如下所示......
Type:Custom, Protocol: TCP, Port Range: 38839, Destination: 0.0.0.0/0
关于mongodb - pymongo.errors.ConnectionFailure : timed out from an ubuntu ec2 instance running scrapyd，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/26964640/