我有一个 from selenium import webdriver
温度网络抓取工具,工作于 weather.com使用Python。页面开头的网络抓取工具几乎立即找到正确的高温和低温。然而,到页面末尾时,它会变得越来越慢(需要 7 秒才能结束)。这可能是因为抓取工具必须遍历更多 HTML 才能找到正确的数据(?)。这是代码的主要部分:
high = driver.find_element_by_css_selector('#twc-scrollabe > table > tbody > tr:nth-child(' + str(j) + ') > td.temp > div > span:nth-child(1)').text
low = driver.find_element_by_css_selector('#twc-scrollabe > table > tbody > tr:nth-child(' + str(j) + ') > td.temp > div > span:nth-child(3)').text
date = driver.find_element_by_css_selector('#twc-scrollabe > table > tbody > tr:nth-child(' + str(j) + ') > td:nth-child(2) > div > span').text
#auth > div > div > div > div > div > form > button
#twc-scrollabe > table > tbody > tr:nth-child(1) > td:nth-child(2) > div > span
#twc-scrollabe > table > tbody > tr:nth-child(2) > td:nth-child(2) > div > span
是否有简单(或复杂)的解决方案?如果您认为没有简单的解决方案,那也会有很大的帮助(可能是这种情况?)
最佳答案
您要查找的内容是否是由 JavaScript 生成的?如果只是 HTML,您可以避免使用 headless 浏览器并使用 requests
和 bs4
:
$ python test.py
Got response: 200
Today JUN 1 80°/61°
Sun JUN 2 70°/47°
Mon JUN 3 63°/45°
Tue JUN 4 74°/57°
Wed JUN 5 75°/64°
Thu JUN 6 77°/63°
Fri JUN 7 77°/64°
Sat JUN 8 81°/66°
Sun JUN 9 81°/65°
Mon JUN 10 80°/63°
Tue JUN 11 80°/63°
Wed JUN 12 81°/62°
Thu JUN 13 80°/63°
Fri JUN 14 81°/63°
Sat JUN 15 81°/63°
Total: 0.66s, request: 0.60s
测试.py
import requests
import time
from bs4 import BeautifulSoup
URL = 'https://weather.com/weather/tenday/l/USPA1290:1:US'
def fetch(url):
with requests.Session() as s:
r = s.get(URL, timeout=5)
return r
def main():
start_t = time.time()
resp = fetch(URL)
print(f'Got response: {resp.status_code}')
html = resp.text
bs = BeautifulSoup(html, 'html.parser')
tds = bs.find_all('td', class_='twc-sticky-col', attrs={'headers': 'day'})
for td in tds:
date_time = td.find_next('span', class_='date-time')
day_detail = td.find_next('span', class_='day-detail')
temp = td.find_next('td', class_='temp', attrs={'headers': 'hi-lo'})
hi_lo = '/'.join(i.text for i in temp.find_all('span', class_=''))
print(f'{date_time.text:5} {day_detail.text:6} {hi_lo}')
end_t = time.time()
elapsed_t = end_t - start_t
r_time = resp.elapsed.total_seconds()
print(f'Total: {elapsed_t:.2f}s, request: {r_time:.2f}s')
if __name__ == '__main__':
main()
关于python - driver.find_element_by_css_selector 在页面末尾运行缓慢,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56383300/