python - 使用 Python urllib 时出现错误

Question

我在使用 Python 的 urllib 时遇到问题。

这是我试过的代码:

import urllib
s = urllib.urlopen("https://www.mci.ir/web/guest/login")

这是我看到的错误:

Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
     s = urllib.urlopen("https://www.mci.ir/web/guest/login")
   File "C:\Python27\lib\urllib.py", line 86, in urlopen
    return opener.open(url)
  File "C:\Python27\lib\urllib.py", line 207, in open
  return getattr(self, name)(url)
 File "C:\Python27\lib\urllib.py", line 450, in open_https
   return self.http_error(url, fp, errcode, errmsg, headers)
 File "C:\Python27\lib\urllib.py", line 371, in http_error
   result = method(url, fp, errcode, errmsg, headers)
 File "C:\Python27\lib\urllib.py", line 634, in http_error_302
   data)
 File "C:\Python27\lib\urllib.py", line 660, in redirect_internal
   return self.open(newurl)
 File "C:\Python27\lib\urllib.py", line 207, in open
   return getattr(self, name)(url)
 File "C:\Python27\lib\urllib.py", line 436, in open_https
   h.endheaders(data)
 File "C:\Python27\lib\httplib.py", line 954, in endheaders
   self._send_output(message_body)
 File "C:\Python27\lib\httplib.py", line 814, in _send_output
   self.send(msg)
 File "C:\Python27\lib\httplib.py", line 776, in send
   self.connect()
 File "C:\Python27\lib\httplib.py", line 1161, in connect
   self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
 File "C:\Python27\lib\ssl.py", line 381, in wrap_socket
   ciphers=ciphers)
 File "C:\Python27\lib\ssl.py", line 143, in __init__
   self.do_handshake()
 File "C:\Python27\lib\ssl.py", line 305, in do_handshake
   self._sslobj.do_handshake()
IOError: [Errno socket error] [Errno 8] _ssl.c:504: EOF occurred in         violation of protocol

Answer 1

远程服务器似乎不喜欢 urllib.urlopen() 和 urllib2.urlopen() 使用的 User-Agent header >(Python 2)，也不是 urllib.request.urlopen()(Python 3)。它正在关闭连接。

使用 requests 发出请求包确实有效:

>>> import requests
>>> r = requests.get('https://www.mci.ir/web/guest/login')
>>> r
<Response [200]>

将用户代理设置为 urllib/urllib2 使用的用户代理:

>>> r = requests.get('https://www.mci.ir/web/guest/login', headers={'User-Agent': 'Python-urllib/2.7'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/sessions.py", line 594, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/sessions.py", line 196, in resolve_redirects
    **adapter_kwargs
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/mhawke/virtualenvs/py2/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)

我的建议是使用 requests 因为这是一个更好的库，但是，如果您必须使用标准库，请使用 urllib2 并设置一个用户代理头远程服务器可以接受:

req = urllib2.Request('https://www.mci.ir/web/guest/login')
req.add_header('User-Agent','Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0')
r = urllib2.urlopen(req)
html = r.read()

另一件值得注意的事情是，一旦远程服务器收到它不喜欢的请求(例如，使用未接受的用户代理)，它会阻止来自原始 IP 地址的请求，直到有一段时间没有请求(或者它可能是一个随机周期)。