python - 美丽汤错误 : '<class ' bs4. 元素。标签'>' object has no attribute ' 内容'？

Question

我正在编写一个脚本，从文章中提取内容并删除任何不必要的内容，例如。脚本和样式。 Beautiful Soup 不断引发以下异常:

'<class 'bs4.element.Tag'>' object has no attribute 'contents'

以下是trim函数的代码(element是包含网页内容的HTML元素):

def trim(element):
    elements_to_remove = ('script', 'style', 'link', 'form', 'object', 'iframe')
    for i in elements_to_remove:
        remove_all_elements(element, i)

    attributes_to_remove = ('class', 'id', 'style')
    for i in attributes_to_remove:
        remove_all_attributes(element, i)

    remove_all_comments(element)

    # Remove divs that have more non-p elements than p elements
    for div in element.find_all('div'):
        p = len(div.find_all('p'))
        img = len(div.find_all('img'))
        li = len(div.find_all('li'))
        a = len(div.find_all('a'))

        if p == 0 or img > p or li > p or a > p:
            div.decompose()

查看堆栈跟踪，问题似乎出在 for 语句之后的此方法:

    # Remove divs that have more non-p elements than p elements
    for div in element.find_all('div'):
        p = len(div.find_all('p')) # <-- div.find_all('p')

我不明白为什么 bs4.element.Tag 的这个实例没有“contents”属性？我在实际网页上尝试了一下，元素中充满了 p 和 img...

这是回溯(这是我正在开发的 Django 项目的一部分):

Environment:


Request Method: POST
Request URL: http://localhost:8000/read/add/

Django Version: 1.4.1
Python Version: 2.7.3
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'home',
 'account',
 'read',
 'review')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Traceback:
File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/marco/sandra/read/views.py" in add
  24.             Article.objects.create_article(request.user, url)
File "/home/marco/sandra/read/models.py" in create_article
  11.         title, content = logic.process_html(web_page.read())
File "/home/marco/sandra/read/logic.py" in process_html
  7.     soup = htmlbarber.give_haircut(BeautifulSoup(html_code, 'html5lib'))
File "/home/marco/sandra/read/htmlbarber/__init__.py" in give_haircut
  45.     scissor.trim(element)
File "/home/marco/sandra/read/htmlbarber/scissor.py" in trim
  35.         p = len(div.find_all('p'))
File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in find_all
  1128.         return self._find_all(name, attrs, text, limit, generator, **kwargs)
File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in _find_all
  413.                 return [element for element in generator
File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in descendants
  1140.         if not len(self.contents):
File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in __getattr__
  924.             "'%s' object has no attribute '%s'" % (self.__class__, tag))

Exception Type: AttributeError at /read/add/
Exception Value: '<class 'bs4.element.Tag'>' object has no attribute 'contents'

这是remove_all_*函数的源代码:

def remove_all_elements(element_to_clean, unwanted_element_name):
    for to_remove in element_to_clean.find_all(unwanted_element_name):
        to_remove.decompose()

def remove_all_attributes(element_to_clean, unwanted_attribute_name):
    for to_inspect in [element_to_clean] + element_to_clean.find_all():
        try:
            del to_inspect[unwanted_attribute_name]
        except KeyError:
            pass

def remove_all_comments(element_to_clean):
    for comment in element_to_clean.find_all(text=lambda text:isinstance(text, Comment)):
        comment.extract()

Answer 1

我认为问题在于，在 remove_all_elements 或代码中的其他位置，您正在删除某些标记的 contents 属性。

看起来当您调用to_remove.decompose()时就会发生这种情况。这是该方法的来源:

def decompose(self):
    """Recursively destroys the contents of this tree."""
    self.extract()
    i = self
    while i is not None:
        next = i.next_element
        i.__dict__.clear()
        i = next

如果您手动调用此函数，会发生以下情况:

>> soup = BeautifulSoup('<div><p>hi</p></div>')
>>> d0 = soup.find_all('div')[0]
>>> d0
<div><p>hi</p></div>
>>> d0.decompose()
>>> d0
Traceback (most recent call last):
...
Traceback (most recent call last):
AttributeError: '<class 'bs4.element.Tag'>' object has no attribute 'contents'

看来，一旦您在标签上调用了decompose，您就不能再尝试使用该标签。我不太确定这是在哪里发生的。

我要尝试检查的一件事是，在 trim() 函数中始终保持 len(element.__dict__) > 0 。