python - Scrapy/OpenSSL 抓取 HTTPS 站点 : AttributeError: 'module' object has no attribute 'SSL_CTX_set_session_id_context'

Question

我尝试抓取 HTTPS 网站，但收到以下错误:

2016-05-14 20:11:21 [scrapy] ERROR: Error downloading <GET https://www.anHTTPSsite.com/>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/usr/local/lib/python3.4/dist-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
    return handler.download_request(request, spider)
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/handlers/http11.py", line 60, in download_request
    return agent.download_request(request)
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/handlers/http11.py", line 281, in download_request
    method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/client.py", line 1594, in request
    endpoint = self._getEndpoint(parsedURI)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/client.py", line 1578, in _getEndpoint
    return self._endpointFactory.endpointForURI(uri)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/client.py", line 1454, in endpointForURI
    uri.port)
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/contextfactory.py", line 57, in creatorForNetloc
    return ScrapyClientTLSOptions(hostname.decode("ascii"), self.getContext())
  File "/usr/local/lib/python3.4/dist-packages/scrapy/core/downloader/contextfactory.py", line 54, in getContext
    return self.getCertificateOptions().getContext()
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1618, in getContext
    self._context = self._makeContext()
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/_sslverify.py", line 1656, in _makeContext
    ctx.set_session_id(sessionName)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 719, in set_session_id
    _lib.SSL_CTX_set_session_id_context(
AttributeError: 'module' object has no attribute 'SSL_CTX_set_session_id_context'

但是，当我尝试抓取其 HTTP 版本时(只需将 URL 中的协议(protocol)从 https 更改为 http)，错误就消失了。

编辑:scrapy版本-v输出:

Scrapy    : 1.1.0
lxml      : 3.4.4.0
libxml2   : 2.9.2
Twisted   : 16.1.1
Python    : 3.4.3+ (default, Oct 14 2015, 16:03:50) - [GCC 5.2.1 20151010]
pyOpenSSL : 16.0.0 (OpenSSL 1.0.2d 9 Jul 2015)
Platform  : Linux-4.2.0-36-generic-x86_64-with-Ubuntu-15.10-wily

任何帮助将不胜感激。

Answer 1

我也遇到了同样的问题。它似乎与 pyOpenSSL 最近的更改有关。降级到 0.15.1 为我解决了这个问题。在此之前，我建议您查看 pyOpenSSL 的变更日志，看看您是否觉得降级值得。

在 OSX 上，这应该有效:

sudo pip install pyOpenSSL==0.15.1