python - 请求库在 Python 2 和 Python 3 上崩溃

Question

我正在尝试使用以下代码的 requests 和 BeautifulSoup 库解析任意网页:

try:
    response = requests.get(url)
except Exception as error:
    return False

if response.encoding == None:
    soup = bs4.BeautifulSoup(response.text) # This is line 809
else:
    soup = bs4.BeautifulSoup(response.text, from_encoding=response.encoding)

在大多数网页上，这都可以正常工作。但是，在某些任意页面 (<1%) 上我遇到此崩溃:

Traceback (most recent call last):
  File "/home/dotancohen/code/parser.py", line 155, in has_css
    soup = bs4.BeautifulSoup(response.text)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 809, in text
    content = str(self.content, encoding, errors='replace')
  TypeError: str() argument 2 must be str, not None

作为引用，这是requests库的相关方法:

@property
def text(self):
    """Content of the response, in unicode.

    if Response.encoding is None and chardet module is available, encoding
    will be guessed.
    """

    # Try charset from content-type
    content = None
    encoding = self.encoding

    # Fallback to auto-detected encoding.
    if self.encoding is None:
        if chardet is not None:
            encoding = chardet.detect(self.content)['encoding']

    # Decode unicode from given encoding.
    try:
        content = str(self.content, encoding, errors='replace') # This is line 809
    except LookupError:
        # A LookupError is raised if the encoding was not found which could
        # indicate a misspelling or similar mistake.
        #
        # So we try blindly encoding.
        content = str(self.content, errors='replace')

    return content

可以看出，当抛出此错误时，我没有传递编码。 我如何错误地使用该库，我该怎么做才能防止此错误？这是在 Python 3.2.3 上进行的，但我也可以使用 Python 2 获得相同的结果。

Answer 1

这意味着服务器没有发送 header 中内容的编码，并且 chardet 库也无法确定内容的编码。事实上，您故意测试是否缺少编码；如果没有可用的编码，为什么要尝试获取解码的文本？

您可以尝试将解码留给 BeautifulSoup 解析器:

if response.encoding is None:
   soup = bs4.BeautifulSoup(response.content)

并且不需要将编码传递给BeautifulSoup，因为如果.text没有失败，那么您正在使用Unicode，BeautifulSoup无论如何都会忽略编码参数:

else:
   soup = bs4.BeautifulSoup(response.text)