这是我的代码:
import requests as req
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Kevin\'s_request'}
r = req.get('https://www.amazon.com/Python-Crash-Course-Hands-Project-Based/dp/1593276036/ref=sr_1_3?ie=UTF8&qid=1541450645&sr=8-3&keywords=python', headers=headers)
soup = BeautifulSoup(r.text, "html.parser")
soup.find(class_="a-expander-content a-expander-partial-collapse-content")
我最终只得到一个空列表。我在 Jupyter Notebooks 和 BS 4 中使用 Python 3.6.4
最佳答案
尝试这种方法。结果你的选择器找不到任何东西。不过,我已经修复了它以达到目的:
import requests
from bs4 import BeautifulSoup
def get_reviews(s,url):
s.headers['User-Agent'] = 'Mozilla/5.0'
response = s.get(url)
soup = BeautifulSoup(response.text,"lxml")
return soup.find_all("div",{"data-hook":"review-collapsed"})
if __name__ == '__main__':
link = 'https://www.amazon.com/Python-Crash-Course-Hands-Project-Based/dp/1593276036/ref=sr_1_3?ie=UTF8&qid=1541450645&sr=8-3&keywords=python'
with requests.Session() as s:
for review in get_reviews(s,link):
print(f'{review.text}\n')
关于python - 使用 BeautifulSoup 抓取亚马逊网页,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/53384422/