我正在尝试从 this 中抓取文本网站。它返回这样的文本:
डा. à¤à¥à¤·à¤¬à¤¹à¤¾à¤¦à¥à¤° थापालाठपà¥à¤¤à¥à¤°à¥à¤¶à¥à¤, à¤à¤®à¥à¤°à¤¿à¤à¤¾à¤®à¤¾ तà¥à¤à¤¶à¥à¤°à¥à¤à¥ निधन
而不是:
भारतीय विदेश सचिव गोखले आज नेपाल आउँदै.
当前代码:
headers = {
'Connection': 'close',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
}
def get_url_soup(url):
url_request = requests.get(url, headers=headers, allow_redirects=True)
soup = BeautifulSoup(url_request.text, 'lxml')
return soup
soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})
最佳答案
使用EncodingDetector :
from bs4.dammit import EncodingDetector
headers = {
'Connection': 'close',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
}
def get_url_soup(url):
url_request = requests.get(url, headers=headers, allow_redirects=True)
http_encoding = url_request.encoding if 'charset' in url_request.headers.get('content-type', '').lower() else None
html_encoding = EncodingDetector.find_declared_encoding(url_request.content, is_html=True)
encoding = html_encoding or http_encoding
soup = BeautifulSoup(url_request.content, 'lxml', from_encoding=encoding)
return soup
soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})
print(title_card.text)
输出:
होमपेज /
समाचार /
राष्ट्रिय समाचार
भारतीय विदेश सचिव गोखले आज नेपाल आउँदै
प्रधानमन्त्रीलगायत शीर्ष नेतासँग भेट्ने
.
.
.
关于python - 使用 Beautiful Soup 抓取网站时出现编码错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/55391495/