这是经过几个小时的抓取后的错误回溯:
The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.
这是我对 selenium python 的设置:
#scrape.py
from selenium.common.exceptions import *
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
def run_scrape(link):
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--headless")
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("--lang=en")
chrome_options.add_argument("--start-maximized")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36")
chrome_options.binary_location = "/usr/bin/google-chrome"
browser = webdriver.Chrome(executable_path=r'/usr/local/bin/chromedriver', options=chrome_options)
browser.get(<link passed here>)
try:
#scrape process
except:
#other stuffs
browser.quit()
#multiprocess.py
import time,
from multiprocessing import Pool
from scrape import *
if __name__ == '__main__':
start_time = time.time()
#links = list of links to be scraped
pool = Pool(20)
results = pool.map(run_scrape, links)
pool.close()
print("Total Time Processed: "+"--- %s seconds ---" % (time.time() - start_time))
Chrome、ChromeDriver 设置、Selenium 版本
ChromeDriver 79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-heads/3945@{#614})
Google Chrome 79.0.3945.79
Selenium Version: 4.0.0a3
我想知道为什么 chrome 正在关闭,但其他进程正在运行?
最佳答案
我拿了你的代码,对其进行了一些修改以适应我的测试环境,这是执行结果:
代码块:
multiprocess.py:
import time from multiprocessing import Pool from multiprocessingPool.scrape import run_scrape if __name__ == '__main__': start_time = time.time() links = ["https://selenium.dev/downloads/", "https://selenium.dev/documentation/en/"] pool = Pool(2) results = pool.map(run_scrape, links) pool.close() print("Total Time Processed: "+"--- %s seconds ---" % (time.time() - start_time))
scrape.py:
from selenium import webdriver from selenium.common.exceptions import NoSuchElementException, TimeoutException from selenium.webdriver.common.by import By from selenium.webdriver.chrome.options import Options def run_scrape(link): chrome_options = Options() chrome_options.add_argument('--no-sandbox') chrome_options.add_argument("--headless") chrome_options.add_argument('--disable-dev-shm-usage') chrome_options.add_argument("--lang=en") chrome_options.add_argument("--start-maximized") chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"]) chrome_options.add_experimental_option('useAutomationExtension', False) chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36") chrome_options.binary_location=r'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe' browser = webdriver.Chrome(executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe', options=chrome_options) browser.get(link) try: print(browser.title) except (NoSuchElementException, TimeoutException): print("Error") browser.quit()
控制台输出:
Downloads The Selenium Browser Automation Project :: Documentation for Selenium Total Time Processed: --- 10.248600006103516 seconds ---
结论
很明显,您的程序在逻辑上是完美无缺的。
这个用例
正如您提到的,经过几个小时的抓取后出现此错误,我怀疑这是因为 WebDriver is not thread-safe 。话虽如此,如果您可以序列化对底层驱动程序实例的访问,则可以在多个线程中共享引用。这是不可取的。但您始终可以实例化一个WebDriver每个线程的实例。
理想情况下,线程安全问题不在您的代码中,而在实际的浏览器绑定(bind)中。他们都假设一次只有一个命令(例如,像真正的用户一样)。但另一方面,您始终可以为每个线程实例化一个 WebDriver 实例,这将启动多个浏览选项卡/窗口。到目前为止,您的程序似乎很完美。
现在,不同的线程可以在同一个Webdriver上运行,但是测试结果将不是您所期望的。背后的原因是,当您使用多线程在不同的选项卡/窗口上运行不同的测试时,需要一点线程安全编码,否则您将执行诸如 click()
或 之类的操作>send_keys()
将转到当前具有焦点的已打开选项卡/窗口,无论您希望运行哪个线程。这本质上意味着所有测试将在具有焦点但不在目标选项卡/窗口的同一个选项卡/窗口上同时运行。
关于python - 通过 Python 使用 Selenium 进行多处理时,Chrome 在几个小时后崩溃,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/60181613/