html - BeautifulSoup 找不到元素

Question

我开始使用 BeautifulSoup，不幸的是它没有按预期工作。

在以下链接中https://www.globes.co.il/news/article.aspx?did=1001285059包括以下元素:

<div class="sppre_message-data-wrapper">... </div>

我试图通过编写以下代码来获取此元素:

html = urlopen("https://www.globes.co.il/news/article.aspx?did=1001285059")
bsObj = BeautifulSoup(html.read(), features="html.parser")
comments = bsObj.find_all('div', {'class': ["sppre_message-data-wrapper"]})
print(comments)

'comments' 给出了一个空数组

Answer 1

它在 iframe 中。向 iframe src 提出请求

https://spoxy-shard2.spot.im/v2/spot/sp_8BE2orzs/post/1001285059/?elementId=6a97624752c75d958352037d2b36df77&spot_im_platform=desktop&host_url=https%3A%2F%2Fwww.globes.co.il%2Fnews%2Farticle.aspx%3Fdid%3D1001285059&host_url_64=aHR0cHM6Ly93d3cuZ2xvYmVzLmNvLmlsL25ld3MvYXJ0aWNsZS5hc3B4P2RpZD0xMDAxMjg1MDU5&pageSize=1&count=1&spot_im_ph__prerender_deferred=true&prerenderDeferred=true&sort_by=newest&conversationSkin=light&isStarsRatingEnabled=false&enableMessageShare=true&enableAnonymize=true&isConversationLiveBlog=false&enableSeeMoreButton=true

是

from bs4 import BeautifulSoup as bs
import requests

r = requests.get('https://spoxy-shard2.spot.im/v2/spot/sp_8BE2orzs/post/1001285059/?elementId=6a97624752c75d958352037d2b36df77&spot_im_platform=desktop&host_url=https%3A%2F%2Fwww.globes.co.il%2Fnews%2Farticle.aspx%3Fdid%3D1001285059&host_url_64=aHR0cHM6Ly93d3cuZ2xvYmVzLmNvLmlsL25ld3MvYXJ0aWNsZS5hc3B4P2RpZD0xMDAxMjg1MDU5&pageSize=1&count=1&spot_im_ph__prerender_deferred=true&prerenderDeferred=true&sort_by=newest&conversationSkin=light&isStarsRatingEnabled=false&enableMessageShare=true&enableAnonymize=true&isConversationLiveBlog=false&enableSeeMoreButton=true')
soup= bs(r.content,'html.parser')
comments = [item.text for item in soup.select('.sppre_message-data-wrapper')]
print(comments)

BeautifulSoup 不支持深度组合器(我想它现在已经退休了)但是你可以在浏览器(Chrome)中看到它使用:

*/deep/.sppre_message-data-wrapper

最终没有关系，因为原始 url 的请求响应中不存在内容。

我猜您也可以使用 selenium 并切换到 iframe。虽然有一个 ID 为 401bccf8039377de3e9873905037a855-iframe即 find_element_by_css_selector 的 #401bccf8039377de3e9873905037a855-iframe，然后切换到一个更健壮的(如果 id 是动态的)选择器将是 .sppre_frame-container iframe